Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................................X..X.................X] duration: 5.001064844s trace: https://jaeger.dev.storj.io/trace/56a56be8a26ec8d8 FAIL (2025-11-01 17:31:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001105201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................X..X.................] duration: 1.648321151s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:30:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.648336187s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................X..X................] duration: 1.61792286s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:29:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.617950846s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X..X...............] duration: 1.649479067s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:28:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.649569587s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................X..X..............] duration: 1.884781295s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:28:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.884804414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................X..X.............] duration: 1.658193494s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:27:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.658213208s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................X..X............] duration: 1.704080072s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:26:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.704112837s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................X..X...........] duration: 2.263239704s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:25:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.263287952s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................X..X..........] duration: 1.554799007s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:25:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.554822454s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................X..X.........] duration: 1.645767902s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:24:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.645788109s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................X..X........] duration: 1.690760744s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:23:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.690807297s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................................X..X.......] duration: 1.664693906s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:22:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.664725654s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................X..X......] duration: 1.610666782s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:22:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.610698621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................................X..X.....] duration: 1.656697959s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:21:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.656725689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................................X..X....] duration: 2.687015567s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:20:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.687059211s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................................X..X...] duration: 1.615428925s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:20:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.615454799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................................X..X..] duration: 2.179669351s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:19:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.179696885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................................X..X.] duration: 1.531296019s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:18:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.531323907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................................X..X] duration: 5.001121037s trace: https://jaeger.dev.storj.io/trace/dfe801ab14abbf1 FAIL (2025-11-01 17:18:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001144043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 1.623473014s trace: https://jaeger.dev.storj.io/trace/6cd46d15b56ad4b5 FAIL (2025-11-01 17:17:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.623500557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 1.637169689s trace: https://jaeger.dev.storj.io/trace/6cd46d15b56ad4b5 FAIL (2025-11-01 17:17:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.637189729s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.000886097s trace: https://jaeger.dev.storj.io/trace/6cd46d15b56ad4b5 FAIL (2025-11-01 17:16:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000926211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.243511672s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:24:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.243538172s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.282131349s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:23:20 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.282160181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.215901779s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:22:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.215928363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.332835449s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:22:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.332867311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.199963759s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:21:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.199988182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.353383091s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:21:41 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.353399352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.243420272s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:20:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.243445904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.391008837s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:19:40 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.391042484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.322126813s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:19:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.322159517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.331146727s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:18:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.331163818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.357303759s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:18:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.357335166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.425507118s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:17:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.425524396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.271652791s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:15:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.271686867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.32610065s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:15:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.326127881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.321009893s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:14:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.321051275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.220532376s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:13:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.220574359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.227380194s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:13:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.227410429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.18033819s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:12:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.180372566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.221661374s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:11:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.221681406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.332614581s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:10:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.332637064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.310463933s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:10:16 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.31049442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.242946937s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:09:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.242987505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.170571396s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:08:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.170595207s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.342108384s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:07:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.342133865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.285719489s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:06:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.285749334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.274665746s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:06:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.274692058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.255350259s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:05:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.255378298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.375859305s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:04:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.375910623s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)