Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.381214811s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:54:03 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.381243423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.242428716s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:53:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.242454512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.243332571s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:53:16 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.24335934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.316118135s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:52:13 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.316146546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.353155219s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:51:25 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.353183819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.256648353s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:50:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.256679267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.241125635s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:49:20 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.241152196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.260621073s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:48:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.260643625s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.323756863s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:47:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.323787073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.327570404s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:46:06 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.327612008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.30959872s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:45:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.309623236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.279563586s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:44:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 2.279584357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002199662s trace: https://jaeger.dev.storj.io/trace/6a86f0eff9cf537c FAIL (2025-11-01 13:44:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== NT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.907657022 (50h2m24.60924948s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908209212 (50h2m24.609801672s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.908822908 (50h2m24.610415366s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909330058 (50h2m24.610922515s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.909896649 (50h2m24.611489108s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.910627778 (50h2m24.612220233s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911224920 (50h2m24.612817373s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.911631122 (50h2m24.61322357s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912114062 (50h2m24.613706515s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.912451686 (50h2m24.614044126s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913182862 (50h2m24.614775312s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.913651158 (50h2m24.615243623s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914111857 (50h2m24.615704304s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.914436000 (50h2m24.616028458s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:21.915059090 (50h2m24.616651541s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Uploading piece... Nov 1 13:43:22.460576963 (50h2m25.162169415s) [upload=44008]: Next returned nil segment Nov 1 13:43:22.460595398 (50h2m25.162187861s) [upload=44008]: Waiting for error group managing segments... Nov 1 13:43:22.611449731 (50h2m25.313042182s) [noise=true , address=145.79.227.34:4911 , piece_id=TLLDJKXAQO3PE4WOH7ZMRYNIE256W4GBNHMUMBFVBHT5FEMB62ZQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.612282286 (50h2m25.313874741s) [noise=true , address=145.79.227.52:4462 , piece_id=ORQU5XUAXSWO5U35DRXXHHYRBNNBKGBCJZND2EUTWKRJI4OHHWBQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.620304305 (50h2m25.321896753s) [noise=true , address=45.91.116.12:4256 , piece_id=PMFRY73NC2YFEAGC4Y7JAF4RFFOZ5EJUS6PR2MKSBVP2DOT5JL2A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.621327878 (50h2m25.322920329s) [noise=true , address=145.79.227.18:4016 , piece_id=IWM4BA6O7HDFB2BQLIS42AAQ6BUFKWY7QEJYAQNCIUNXRHWLOZUQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623248269 (50h2m25.324840718s) [noise=true , address=145.79.227.2:4144 , piece_id=KIUZEYULI6FVTCU7MI7XNG4UBUZ2452QBUSXHW5FFA62BTW5QGIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.623882641 (50h2m25.325475089s) [noise=true , address=45.91.116.6:4976 , piece_id=OVCYDRZURVMVM7HZXRJNMJ3DWPG6R7GSYWWDZR3OKJTYPLQLGK5A, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.624318273 (50h2m25.325910722s) [noise=true , address=45.91.116.7:4043 , piece_id=SSMGARCVQGKLHX3JZPUNH6BPYT7E5TKX736RRJPA5T5LBIN2IZ7Q, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.626424151 (50h2m25.328016603s) [noise=true , address=145.79.227.44:4197 , piece_id=GPJUT6EALMGEWZHMYAQAT2KOFHRIBCZDGP74FU5IW4JAN6LPBMQQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629396931 (50h2m25.330989381s) [noise=true , address=145.79.227.6:4283 , piece_id=KTQZTF2FIRJLMBO34YC24L6VTWSJ55YDHOZF4SLGPS2QDVF5ZIIA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.629822772 (50h2m25.33141522s) [noise=true , address=77.90.31.18:4810 , piece_id=K5XJ6GJ7N4UL4FWPNC2XITWCOQV427WW6AKTMQDSXFBR4D634GHA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.632940645 (50h2m25.334533093s) [noise=true , address=145.79.227.5:4250 , piece_id=ZY5A2SDU27OYA7R7KKOGRAJLZJNSPUBUIV4KSV5UVW3CU5NDBCMQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.633673271 (50h2m25.335265719s) [noise=true , address=145.79.227.8:4340 , piece_id=H6B5DH6SHBQ4FXZL6DFEW53K7JZ62WVVC5LSRJXF2YKXEKVKZ3VQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.634258067 (50h2m25.335850514s) [noise=true , address=145.79.227.40:4063 , piece_id=XHEH44GA32XDVBOAR425FLLBCW5ULUYYYQCVV4L4OYBHCZISEEVA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639662140 (50h2m25.341254601s) [noise=true , address=145.79.227.41:4110 , piece_id=C6LLQLOYYGJWTRVG2OJPKOYUTBZ3GJX4RXEAUAMDDSLECONTYGDA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: <nil> Nov 1 13:43:22.639691893 (50h2m25.341284351s) [seg_pos={0 0}, upload=44008]: Segment reached optimal threshold of 14 pieces. Nov 1 13:43:22.639973226 (50h2m25.341565683s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36952->64.49.13.169:4706: use of closed network connection Nov 1 13:43:22.639984011 (50h2m25.341576459s) [noise=true , address=64.49.13.169:4706 , piece_id=XH2EVIRZMJ3UXAHWQZFTCTKKGZH4KLKDAPGSTJCEFB35SGRK55MA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640162064 (50h2m25.341754521s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640169275 (50h2m25.341761723s) [noise=true , address=145.79.227.16:4921 , piece_id=WKCM4HBYOYNQVSOLYLLPDBLP3XF4VAOME6W5QBHNTRJJK3K5SEMA, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640240956 (50h2m25.341833399s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G4gW4QhXRDsfHcGvFr7qCpB49u9dYQPDXC4CuBZGwQUuFU7Fx): protocol: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38092->64.49.13.139:4470: use of closed network connection Nov 1 13:43:22.640256358 (50h2m25.341848816s) [noise=true , address=64.49.13.139:4470 , piece_id=74NOS3C3JXLZHR2DDHAIO4WRDCNANYHYLNQG2FFFDJ536IMJAPOQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640274462 (50h2m25.341866898s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UPmXq8zHDsiiL7eNoeUsgkEy1uEfDhPgi816RN41AMdQn1kb5): protocol: expected piece hash; context canceled; EOF Nov 1 13:43:22.640281566 (50h2m25.341874025s) [noise=true , address=145.79.227.46:4624 , piece_id=IBYZ2FA3U7VZBZ6JMAX2OCWOLRQ5GU5JCZYE5MCVUWNT5EKB6ARQ, seg_pos={0 0}, upload=44008]: Long tail cancellation due to optimal threshold Nov 1 13:43:22.640300823 (50h2m25.341893258s) [seg_pos={0 0}, upload=44008]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 1 13:43:22.640385129 (50h2m25.341977584s) [upload=44008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 1 13:43:22.640392321 (50h2m25.341984776s) [upload=44008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 1 13:43:22.780241077 (50h2m25.481833531s) [upload=44008]: Done uploading segments... Nov 1 13:43:22.780262032 (50h2m25.48185449s) [upload=44008]: Upload finished. err: <nil> Duration: 1m0.002225274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.654990715s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 12:03:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.655017748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.61009632s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 12:02:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.610124816s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.664169344s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 12:02:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.664200836s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 1.577627877s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 12:01:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.577643973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 1.658405578s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 12:00:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.658433946s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 1.609063136s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:59:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.609092368s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.531051761s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:59:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.531076931s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.663060314s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:58:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.663087667s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.579924855s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:57:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.579947276s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 1.815375639s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:56:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.815401533s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 1.664128276s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:55:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.66424985s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 1.626463256s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:54:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.626489213s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 1.692584172s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:54:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.69261467s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 1.614154609s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:53:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.614179442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 1.523313218s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:52:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.523342115s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 1.608191492s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:51:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.608224666s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 1.650035869s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:50:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.650064555s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 1.559922659s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:49:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.559945369s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 1.62794963s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:48:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.627976712s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 1.629605078s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:47:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.629628912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 1.588797517s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:46:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.588830971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 1.506740499s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:46:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.506768328s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 1.501666264s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:45:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.50169241s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 1.618198991s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:45:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.618216305s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 1.552142368s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:44:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.552167196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 1.677557949s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:43:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.677585311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 1.575269254s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:43:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.575368673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 1.50946739s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:42:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.50948749s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 1.539541142s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:41:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.539582613s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 1.615933741s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:40:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.615958656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 1.588934866s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:39:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.588961374s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 1.529093456s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:38:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.529119824s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 1.583920934s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:37:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.583978454s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 1.577950411s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:36:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.577976377s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 1.743673838s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:36:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.743701956s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 1.608755502s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:35:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.608775661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 2.138275202s trace: https://jaeger.dev.storj.io/trace/2dd7c1143916b3f2 FAIL (2025-11-01 11:34:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.13836397s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)