Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 1.623473014s trace: https://jaeger.dev.storj.io/trace/6cd46d15b56ad4b5 FAIL (2025-11-01 17:17:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.623500557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 1.637169689s trace: https://jaeger.dev.storj.io/trace/6cd46d15b56ad4b5 FAIL (2025-11-01 17:17:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.637189729s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.000886097s trace: https://jaeger.dev.storj.io/trace/6cd46d15b56ad4b5 FAIL (2025-11-01 17:16:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000926211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.243511672s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:24:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.243538172s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.282131349s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:23:20 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.282160181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.215901779s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:22:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.215928363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.332835449s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:22:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.332867311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.199963759s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:21:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.199988182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.353383091s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:21:41 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.353399352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.243420272s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:20:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.243445904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.391008837s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:19:40 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.391042484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.322126813s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:19:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.322159517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.331146727s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:18:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.331163818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.357303759s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:18:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.357335166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.425507118s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:17:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.425524396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.271652791s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:15:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.271686867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.32610065s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:15:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.326127881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.321009893s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:14:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.321051275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.220532376s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:13:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.220574359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.227380194s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:13:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.227410429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.18033819s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:12:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.180372566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.221661374s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:11:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.221681406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.332614581s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:10:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.332637064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.310463933s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:10:16 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.31049442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.242946937s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:09:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.242987505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.170571396s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:08:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.170595207s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.342108384s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:07:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.342133865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.285719489s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:06:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.285749334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.274665746s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:06:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.274692058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.255350259s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:05:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.255378298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.375859305s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:04:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.375910623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.244334378s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:04:31 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.244360915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.291434686s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:03:40 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.29145983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.35289724s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:03:06 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.352923697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.36593761s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:02:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.365961897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.323687205s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:02:05 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.323718263s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.309604364s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:01:16 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.309626785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.308902688s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 14:00:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.308930409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.282248072s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:59:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.28227242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.354779984s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:59:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.354803098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.318379588s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:58:45 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.318402765s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.294619387s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:58:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.294645471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.271182472s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:58:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.271206239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.260953224s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:57:32 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.260986788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.319234883s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:57:10 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.319260126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.324984278s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:56:46 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.325009872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.263950626s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:56:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.263980682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.337859371s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:56:09 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.337906968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.287745755s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:55:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.2877712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.262298841s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:54:51 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.262328012s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)