Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.147743569s FAIL (2025-07-18 11:54:09 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.147770371s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.850274342s FAIL (2025-07-18 11:53:47 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.850298413s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...............X..........X..............................] duration: 2.387778483s FAIL (2025-07-18 11:53:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.387810829s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X...............X..........X.............................] duration: 2.289893861s FAIL (2025-07-18 11:53:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.289948249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X...............X..........X............................] duration: 2.430467991s FAIL (2025-07-18 11:53:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.430510819s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.520749041s FAIL (2025-07-18 11:52:37 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.520780707s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X...............X..........X...........................] duration: 2.387792537s FAIL (2025-07-18 11:52:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.387823782s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.581330474s FAIL (2025-07-18 11:51:56 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.581377195s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...............X..........X..........................] duration: 2.353964648s FAIL (2025-07-18 11:51:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.353992855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X...............X..........X.........................] duration: 2.311568072s FAIL (2025-07-18 11:51:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.311615719s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.418694652s FAIL (2025-07-18 11:50:43 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.418730357s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X...............X..........X........................] duration: 2.414613161s FAIL (2025-07-18 11:50:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.414674562s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.080377461s FAIL (2025-07-18 11:50:15 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.080404034s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.094227551s FAIL (2025-07-18 11:49:53 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 2.094251204s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X...............X..........X.......................] duration: 2.31612222s FAIL (2025-07-18 11:49:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.316154957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X...............X..........X......................] duration: 3.639715539s FAIL (2025-07-18 11:48:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 3.63974623s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.248097284s FAIL (2025-07-18 11:48:48 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.248124137s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..........X...............X..........X.....................] duration: 2.317378803s FAIL (2025-07-18 11:47:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.317408683s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.050431619s FAIL (2025-07-18 11:47:41 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.050475839s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.085483149s FAIL (2025-07-18 11:47:09 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.085510221s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..........X...............X..........X....................] duration: 2.396686705s FAIL (2025-07-18 11:46:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.396792913s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.01411711s FAIL (2025-07-18 11:46:49 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== HFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567267208 (601h46m53.450680834s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567698610 (601h46m53.451112239s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.570842951 (601h46m53.454256585s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571148694 (601h46m53.454562307s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571617590 (601h46m53.455031203s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572194976 (601h46m53.455608605s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572657235 (601h46m53.45607085s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573103737 (601h46m53.456517363s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573559498 (601h46m53.45697311s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.575230525 (601h46m53.458644152s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.576872259 (601h46m53.460285885s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.578624092 (601h46m53.462037718s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.580779720 (601h46m53.464193352s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.583021682 (601h46m53.466435301s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.584553497 (601h46m53.467967132s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.585102122 (601h46m53.468515755s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.586584733 (601h46m53.46999835s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:54.118272982 (601h46m54.001686609s) [upload=481170]: Next returned nil segment Jul 18 10:57:54.118290512 (601h46m54.001704132s) [upload=481170]: Waiting for error group managing segments... Jul 18 10:57:54.265678399 (601h46m54.149092018s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.272591913 (601h46m54.156005531s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.273166902 (601h46m54.156580518s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.275724342 (601h46m54.15913796s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278917091 (601h46m54.162330707s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278945616 (601h46m54.162359243s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.283715439 (601h46m54.167129051s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.284438587 (601h46m54.167852208s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.294188062 (601h46m54.1776017s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.434300533 (601h46m54.317714163s) [noise=true , address=45.91.95.3:4022 , piece_id=STAZHFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.435093696 (601h46m54.318507314s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.436583511 (601h46m54.31999714s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.451259425 (601h46m54.334673202s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518816025 (601h46m54.402229644s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518860075 (601h46m54.402273706s) [seg_pos={0 0}, upload=481170]: Segment reached optimal threshold of 14 pieces. Jul 18 10:57:54.519304291 (601h46m54.402717909s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUJcG8EBim135i6mRQQrwSBS5CFqVN5ipdFsGdWYH5Y9CxaUe): protocol: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection Jul 18 10:57:54.519319363 (601h46m54.402732994s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UK3CA8sureHVG8LghUPqYxYQgTY4gY7gaejKaMSBjbpY4kc1M): protocol: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection Jul 18 10:57:54.519384628 (601h46m54.402798246s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c19Vd3TVx59sX7RWYHvY99eMUQRqjJCyryCMZ5NdSdqeDQu9s): protocol: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection Jul 18 10:57:54.519402736 (601h46m54.402816368s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: write tcp 139.59.144.22:58638->77.90.31.15:4704: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 10:57:54.519432414 (601h46m54.402846045s) [seg_pos={0 0}, upload=481170]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 18 10:57:54.519523054 (601h46m54.40293687s) [upload=481170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 10:57:54.519529627 (601h46m54.402943243s) [upload=481170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 10:57:54.662927129 (601h46m54.546340748s) [upload=481170]: Done uploading segments... Jul 18 10:57:54.662950246 (601h46m54.546363866s) [upload=481170]: Upload finished. err: <nil> Duration: 3.014152422s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.235323208s FAIL (2025-07-18 11:46:30 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.235395312s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X..........X...............X..........X...................] duration: 2.566858735s FAIL (2025-07-18 11:46:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.566901084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.X..........X...............X..........X..................] duration: 2.358003457s FAIL (2025-07-18 11:46:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.358040205s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.145217269s FAIL (2025-07-18 11:45:51 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.145249667s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.818569205s FAIL (2025-07-18 11:45:36 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== HFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567267208 (601h46m53.450680834s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567698610 (601h46m53.451112239s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.570842951 (601h46m53.454256585s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571148694 (601h46m53.454562307s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571617590 (601h46m53.455031203s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572194976 (601h46m53.455608605s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572657235 (601h46m53.45607085s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573103737 (601h46m53.456517363s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573559498 (601h46m53.45697311s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.575230525 (601h46m53.458644152s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.576872259 (601h46m53.460285885s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.578624092 (601h46m53.462037718s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.580779720 (601h46m53.464193352s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.583021682 (601h46m53.466435301s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.584553497 (601h46m53.467967132s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.585102122 (601h46m53.468515755s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.586584733 (601h46m53.46999835s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:54.118272982 (601h46m54.001686609s) [upload=481170]: Next returned nil segment Jul 18 10:57:54.118290512 (601h46m54.001704132s) [upload=481170]: Waiting for error group managing segments... Jul 18 10:57:54.265678399 (601h46m54.149092018s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.272591913 (601h46m54.156005531s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.273166902 (601h46m54.156580518s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.275724342 (601h46m54.15913796s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278917091 (601h46m54.162330707s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278945616 (601h46m54.162359243s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.283715439 (601h46m54.167129051s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.284438587 (601h46m54.167852208s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.294188062 (601h46m54.1776017s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.434300533 (601h46m54.317714163s) [noise=true , address=45.91.95.3:4022 , piece_id=STAZHFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.435093696 (601h46m54.318507314s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.436583511 (601h46m54.31999714s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.451259425 (601h46m54.334673202s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518816025 (601h46m54.402229644s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518860075 (601h46m54.402273706s) [seg_pos={0 0}, upload=481170]: Segment reached optimal threshold of 14 pieces. Jul 18 10:57:54.519304291 (601h46m54.402717909s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUJcG8EBim135i6mRQQrwSBS5CFqVN5ipdFsGdWYH5Y9CxaUe): protocol: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection Jul 18 10:57:54.519319363 (601h46m54.402732994s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UK3CA8sureHVG8LghUPqYxYQgTY4gY7gaejKaMSBjbpY4kc1M): protocol: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection Jul 18 10:57:54.519384628 (601h46m54.402798246s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c19Vd3TVx59sX7RWYHvY99eMUQRqjJCyryCMZ5NdSdqeDQu9s): protocol: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection Jul 18 10:57:54.519402736 (601h46m54.402816368s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: write tcp 139.59.144.22:58638->77.90.31.15:4704: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 10:57:54.519432414 (601h46m54.402846045s) [seg_pos={0 0}, upload=481170]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 18 10:57:54.519523054 (601h46m54.40293687s) [upload=481170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 10:57:54.519529627 (601h46m54.402943243s) [upload=481170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 10:57:54.662927129 (601h46m54.546340748s) [upload=481170]: Done uploading segments... Jul 18 10:57:54.662950246 (601h46m54.546363866s) [upload=481170]: Upload finished. err: <nil> Duration: 2.818597112s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.049110408s FAIL (2025-07-18 11:45:26 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.04913369s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.X..........X...............X..........X.................] duration: 2.320872689s FAIL (2025-07-18 11:45:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.320911805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.X..........X...............X..........X................] duration: 2.432583301s FAIL (2025-07-18 11:44:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.432612969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.821551557s FAIL (2025-07-18 11:44:35 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== HFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567267208 (601h46m53.450680834s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567698610 (601h46m53.451112239s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.570842951 (601h46m53.454256585s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571148694 (601h46m53.454562307s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571617590 (601h46m53.455031203s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572194976 (601h46m53.455608605s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572657235 (601h46m53.45607085s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573103737 (601h46m53.456517363s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573559498 (601h46m53.45697311s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.575230525 (601h46m53.458644152s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.576872259 (601h46m53.460285885s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.578624092 (601h46m53.462037718s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.580779720 (601h46m53.464193352s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.583021682 (601h46m53.466435301s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.584553497 (601h46m53.467967132s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.585102122 (601h46m53.468515755s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.586584733 (601h46m53.46999835s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:54.118272982 (601h46m54.001686609s) [upload=481170]: Next returned nil segment Jul 18 10:57:54.118290512 (601h46m54.001704132s) [upload=481170]: Waiting for error group managing segments... Jul 18 10:57:54.265678399 (601h46m54.149092018s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.272591913 (601h46m54.156005531s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.273166902 (601h46m54.156580518s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.275724342 (601h46m54.15913796s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278917091 (601h46m54.162330707s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278945616 (601h46m54.162359243s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.283715439 (601h46m54.167129051s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.284438587 (601h46m54.167852208s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.294188062 (601h46m54.1776017s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.434300533 (601h46m54.317714163s) [noise=true , address=45.91.95.3:4022 , piece_id=STAZHFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.435093696 (601h46m54.318507314s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.436583511 (601h46m54.31999714s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.451259425 (601h46m54.334673202s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518816025 (601h46m54.402229644s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518860075 (601h46m54.402273706s) [seg_pos={0 0}, upload=481170]: Segment reached optimal threshold of 14 pieces. Jul 18 10:57:54.519304291 (601h46m54.402717909s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUJcG8EBim135i6mRQQrwSBS5CFqVN5ipdFsGdWYH5Y9CxaUe): protocol: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection Jul 18 10:57:54.519319363 (601h46m54.402732994s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UK3CA8sureHVG8LghUPqYxYQgTY4gY7gaejKaMSBjbpY4kc1M): protocol: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection Jul 18 10:57:54.519384628 (601h46m54.402798246s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c19Vd3TVx59sX7RWYHvY99eMUQRqjJCyryCMZ5NdSdqeDQu9s): protocol: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection Jul 18 10:57:54.519402736 (601h46m54.402816368s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: write tcp 139.59.144.22:58638->77.90.31.15:4704: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 10:57:54.519432414 (601h46m54.402846045s) [seg_pos={0 0}, upload=481170]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 18 10:57:54.519523054 (601h46m54.40293687s) [upload=481170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 10:57:54.519529627 (601h46m54.402943243s) [upload=481170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 10:57:54.662927129 (601h46m54.546340748s) [upload=481170]: Done uploading segments... Jul 18 10:57:54.662950246 (601h46m54.546363866s) [upload=481170]: Upload finished. err: <nil> Duration: 2.82159117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.X..........X...............X..........X...............] duration: 2.330763947s FAIL (2025-07-18 11:44:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.330795796s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.071712706s FAIL (2025-07-18 11:44:16 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.071735509s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.X..........X...............X..........X..............] duration: 2.312747155s FAIL (2025-07-18 11:43:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.312775674s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.112488222s FAIL (2025-07-18 11:43:44 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.112516516s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 4.19109083s FAIL (2025-07-18 11:43:27 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== HFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567267208 (601h46m53.450680834s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567698610 (601h46m53.451112239s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.570842951 (601h46m53.454256585s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571148694 (601h46m53.454562307s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571617590 (601h46m53.455031203s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572194976 (601h46m53.455608605s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572657235 (601h46m53.45607085s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573103737 (601h46m53.456517363s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573559498 (601h46m53.45697311s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.575230525 (601h46m53.458644152s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.576872259 (601h46m53.460285885s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.578624092 (601h46m53.462037718s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.580779720 (601h46m53.464193352s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.583021682 (601h46m53.466435301s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.584553497 (601h46m53.467967132s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.585102122 (601h46m53.468515755s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.586584733 (601h46m53.46999835s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:54.118272982 (601h46m54.001686609s) [upload=481170]: Next returned nil segment Jul 18 10:57:54.118290512 (601h46m54.001704132s) [upload=481170]: Waiting for error group managing segments... Jul 18 10:57:54.265678399 (601h46m54.149092018s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.272591913 (601h46m54.156005531s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.273166902 (601h46m54.156580518s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.275724342 (601h46m54.15913796s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278917091 (601h46m54.162330707s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278945616 (601h46m54.162359243s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.283715439 (601h46m54.167129051s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.284438587 (601h46m54.167852208s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.294188062 (601h46m54.1776017s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.434300533 (601h46m54.317714163s) [noise=true , address=45.91.95.3:4022 , piece_id=STAZHFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.435093696 (601h46m54.318507314s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.436583511 (601h46m54.31999714s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.451259425 (601h46m54.334673202s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518816025 (601h46m54.402229644s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518860075 (601h46m54.402273706s) [seg_pos={0 0}, upload=481170]: Segment reached optimal threshold of 14 pieces. Jul 18 10:57:54.519304291 (601h46m54.402717909s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUJcG8EBim135i6mRQQrwSBS5CFqVN5ipdFsGdWYH5Y9CxaUe): protocol: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection Jul 18 10:57:54.519319363 (601h46m54.402732994s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UK3CA8sureHVG8LghUPqYxYQgTY4gY7gaejKaMSBjbpY4kc1M): protocol: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection Jul 18 10:57:54.519384628 (601h46m54.402798246s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c19Vd3TVx59sX7RWYHvY99eMUQRqjJCyryCMZ5NdSdqeDQu9s): protocol: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection Jul 18 10:57:54.519402736 (601h46m54.402816368s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: write tcp 139.59.144.22:58638->77.90.31.15:4704: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 10:57:54.519432414 (601h46m54.402846045s) [seg_pos={0 0}, upload=481170]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 18 10:57:54.519523054 (601h46m54.40293687s) [upload=481170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 10:57:54.519529627 (601h46m54.402943243s) [upload=481170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 10:57:54.662927129 (601h46m54.546340748s) [upload=481170]: Done uploading segments... Jul 18 10:57:54.662950246 (601h46m54.546363866s) [upload=481170]: Upload finished. err: <nil> Duration: 4.191120822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.....X.X..........X...............X..........X.............] duration: 2.432463455s FAIL (2025-07-18 11:43:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.432494706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.715854122s FAIL (2025-07-18 11:42:45 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== HFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567267208 (601h46m53.450680834s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567698610 (601h46m53.451112239s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.570842951 (601h46m53.454256585s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571148694 (601h46m53.454562307s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571617590 (601h46m53.455031203s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572194976 (601h46m53.455608605s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572657235 (601h46m53.45607085s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573103737 (601h46m53.456517363s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573559498 (601h46m53.45697311s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.575230525 (601h46m53.458644152s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.576872259 (601h46m53.460285885s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.578624092 (601h46m53.462037718s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.580779720 (601h46m53.464193352s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.583021682 (601h46m53.466435301s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.584553497 (601h46m53.467967132s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.585102122 (601h46m53.468515755s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.586584733 (601h46m53.46999835s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:54.118272982 (601h46m54.001686609s) [upload=481170]: Next returned nil segment Jul 18 10:57:54.118290512 (601h46m54.001704132s) [upload=481170]: Waiting for error group managing segments... Jul 18 10:57:54.265678399 (601h46m54.149092018s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.272591913 (601h46m54.156005531s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.273166902 (601h46m54.156580518s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.275724342 (601h46m54.15913796s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278917091 (601h46m54.162330707s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278945616 (601h46m54.162359243s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.283715439 (601h46m54.167129051s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.284438587 (601h46m54.167852208s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.294188062 (601h46m54.1776017s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.434300533 (601h46m54.317714163s) [noise=true , address=45.91.95.3:4022 , piece_id=STAZHFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.435093696 (601h46m54.318507314s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.436583511 (601h46m54.31999714s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.451259425 (601h46m54.334673202s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518816025 (601h46m54.402229644s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518860075 (601h46m54.402273706s) [seg_pos={0 0}, upload=481170]: Segment reached optimal threshold of 14 pieces. Jul 18 10:57:54.519304291 (601h46m54.402717909s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUJcG8EBim135i6mRQQrwSBS5CFqVN5ipdFsGdWYH5Y9CxaUe): protocol: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection Jul 18 10:57:54.519319363 (601h46m54.402732994s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UK3CA8sureHVG8LghUPqYxYQgTY4gY7gaejKaMSBjbpY4kc1M): protocol: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection Jul 18 10:57:54.519384628 (601h46m54.402798246s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c19Vd3TVx59sX7RWYHvY99eMUQRqjJCyryCMZ5NdSdqeDQu9s): protocol: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection Jul 18 10:57:54.519402736 (601h46m54.402816368s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: write tcp 139.59.144.22:58638->77.90.31.15:4704: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 10:57:54.519432414 (601h46m54.402846045s) [seg_pos={0 0}, upload=481170]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 18 10:57:54.519523054 (601h46m54.40293687s) [upload=481170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 10:57:54.519529627 (601h46m54.402943243s) [upload=481170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 10:57:54.662927129 (601h46m54.546340748s) [upload=481170]: Done uploading segments... Jul 18 10:57:54.662950246 (601h46m54.546363866s) [upload=481170]: Upload finished. err: <nil> Duration: 2.71588996s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.092831091s FAIL (2025-07-18 11:42:36 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.092855063s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.....X.X..........X...............X..........X............] duration: 2.414516452s FAIL (2025-07-18 11:42:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.414540503s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.554610369s FAIL (2025-07-18 11:42:06 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== HFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567267208 (601h46m53.450680834s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567698610 (601h46m53.451112239s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.570842951 (601h46m53.454256585s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571148694 (601h46m53.454562307s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571617590 (601h46m53.455031203s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572194976 (601h46m53.455608605s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572657235 (601h46m53.45607085s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573103737 (601h46m53.456517363s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573559498 (601h46m53.45697311s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.575230525 (601h46m53.458644152s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.576872259 (601h46m53.460285885s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.578624092 (601h46m53.462037718s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.580779720 (601h46m53.464193352s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.583021682 (601h46m53.466435301s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.584553497 (601h46m53.467967132s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.585102122 (601h46m53.468515755s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.586584733 (601h46m53.46999835s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:54.118272982 (601h46m54.001686609s) [upload=481170]: Next returned nil segment Jul 18 10:57:54.118290512 (601h46m54.001704132s) [upload=481170]: Waiting for error group managing segments... Jul 18 10:57:54.265678399 (601h46m54.149092018s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.272591913 (601h46m54.156005531s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.273166902 (601h46m54.156580518s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.275724342 (601h46m54.15913796s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278917091 (601h46m54.162330707s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278945616 (601h46m54.162359243s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.283715439 (601h46m54.167129051s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.284438587 (601h46m54.167852208s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.294188062 (601h46m54.1776017s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.434300533 (601h46m54.317714163s) [noise=true , address=45.91.95.3:4022 , piece_id=STAZHFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.435093696 (601h46m54.318507314s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.436583511 (601h46m54.31999714s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.451259425 (601h46m54.334673202s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518816025 (601h46m54.402229644s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518860075 (601h46m54.402273706s) [seg_pos={0 0}, upload=481170]: Segment reached optimal threshold of 14 pieces. Jul 18 10:57:54.519304291 (601h46m54.402717909s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUJcG8EBim135i6mRQQrwSBS5CFqVN5ipdFsGdWYH5Y9CxaUe): protocol: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection Jul 18 10:57:54.519319363 (601h46m54.402732994s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UK3CA8sureHVG8LghUPqYxYQgTY4gY7gaejKaMSBjbpY4kc1M): protocol: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection Jul 18 10:57:54.519384628 (601h46m54.402798246s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c19Vd3TVx59sX7RWYHvY99eMUQRqjJCyryCMZ5NdSdqeDQu9s): protocol: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection Jul 18 10:57:54.519402736 (601h46m54.402816368s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: write tcp 139.59.144.22:58638->77.90.31.15:4704: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 10:57:54.519432414 (601h46m54.402846045s) [seg_pos={0 0}, upload=481170]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 18 10:57:54.519523054 (601h46m54.40293687s) [upload=481170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 10:57:54.519529627 (601h46m54.402943243s) [upload=481170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 10:57:54.662927129 (601h46m54.546340748s) [upload=481170]: Done uploading segments... Jul 18 10:57:54.662950246 (601h46m54.546363866s) [upload=481170]: Upload finished. err: <nil> Duration: 2.55464345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.18740871s FAIL (2025-07-18 11:41:58 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.187430847s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.59306206s FAIL (2025-07-18 11:41:21 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== HFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567267208 (601h46m53.450680834s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567698610 (601h46m53.451112239s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.570842951 (601h46m53.454256585s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571148694 (601h46m53.454562307s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571617590 (601h46m53.455031203s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572194976 (601h46m53.455608605s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572657235 (601h46m53.45607085s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573103737 (601h46m53.456517363s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573559498 (601h46m53.45697311s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.575230525 (601h46m53.458644152s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.576872259 (601h46m53.460285885s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.578624092 (601h46m53.462037718s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.580779720 (601h46m53.464193352s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.583021682 (601h46m53.466435301s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.584553497 (601h46m53.467967132s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.585102122 (601h46m53.468515755s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.586584733 (601h46m53.46999835s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:54.118272982 (601h46m54.001686609s) [upload=481170]: Next returned nil segment Jul 18 10:57:54.118290512 (601h46m54.001704132s) [upload=481170]: Waiting for error group managing segments... Jul 18 10:57:54.265678399 (601h46m54.149092018s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.272591913 (601h46m54.156005531s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.273166902 (601h46m54.156580518s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.275724342 (601h46m54.15913796s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278917091 (601h46m54.162330707s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278945616 (601h46m54.162359243s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.283715439 (601h46m54.167129051s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.284438587 (601h46m54.167852208s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.294188062 (601h46m54.1776017s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.434300533 (601h46m54.317714163s) [noise=true , address=45.91.95.3:4022 , piece_id=STAZHFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.435093696 (601h46m54.318507314s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.436583511 (601h46m54.31999714s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.451259425 (601h46m54.334673202s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518816025 (601h46m54.402229644s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518860075 (601h46m54.402273706s) [seg_pos={0 0}, upload=481170]: Segment reached optimal threshold of 14 pieces. Jul 18 10:57:54.519304291 (601h46m54.402717909s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUJcG8EBim135i6mRQQrwSBS5CFqVN5ipdFsGdWYH5Y9CxaUe): protocol: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection Jul 18 10:57:54.519319363 (601h46m54.402732994s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UK3CA8sureHVG8LghUPqYxYQgTY4gY7gaejKaMSBjbpY4kc1M): protocol: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection Jul 18 10:57:54.519384628 (601h46m54.402798246s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c19Vd3TVx59sX7RWYHvY99eMUQRqjJCyryCMZ5NdSdqeDQu9s): protocol: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection Jul 18 10:57:54.519402736 (601h46m54.402816368s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: write tcp 139.59.144.22:58638->77.90.31.15:4704: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 10:57:54.519432414 (601h46m54.402846045s) [seg_pos={0 0}, upload=481170]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 18 10:57:54.519523054 (601h46m54.40293687s) [upload=481170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 10:57:54.519529627 (601h46m54.402943243s) [upload=481170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 10:57:54.662927129 (601h46m54.546340748s) [upload=481170]: Done uploading segments... Jul 18 10:57:54.662950246 (601h46m54.546363866s) [upload=481170]: Upload finished. err: <nil> Duration: 2.593092486s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.137928427s FAIL (2025-07-18 11:41:10 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.137950135s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.....X.X..........X...............X..........X...........] duration: 2.460314763s FAIL (2025-07-18 11:41:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.460344857s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.....X.X..........X...............X..........X..........] duration: 2.49398221s FAIL (2025-07-18 11:40:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.49401287s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.078018309s FAIL (2025-07-18 11:40:38 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.078043779s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.....X.X..........X...............X..........X.........] duration: 2.457111439s FAIL (2025-07-18 11:40:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.457139377s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.655514826s FAIL (2025-07-18 11:40:19 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== HFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567267208 (601h46m53.450680834s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.567698610 (601h46m53.451112239s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.570842951 (601h46m53.454256585s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571148694 (601h46m53.454562307s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.571617590 (601h46m53.455031203s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572194976 (601h46m53.455608605s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.572657235 (601h46m53.45607085s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573103737 (601h46m53.456517363s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.573559498 (601h46m53.45697311s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.575230525 (601h46m53.458644152s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.576872259 (601h46m53.460285885s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.578624092 (601h46m53.462037718s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.580779720 (601h46m53.464193352s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.583021682 (601h46m53.466435301s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.584553497 (601h46m53.467967132s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.585102122 (601h46m53.468515755s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:53.586584733 (601h46m53.46999835s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Uploading piece... Jul 18 10:57:54.118272982 (601h46m54.001686609s) [upload=481170]: Next returned nil segment Jul 18 10:57:54.118290512 (601h46m54.001704132s) [upload=481170]: Waiting for error group managing segments... Jul 18 10:57:54.265678399 (601h46m54.149092018s) [noise=true , address=77.90.31.19:4849 , piece_id=2MY5DNKS7AAE3ME4SJ5XWXHGAICES3VH5SP4ADDJXEACDNCFUK7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.272591913 (601h46m54.156005531s) [noise=true , address=77.90.31.16:4760 , piece_id=SLUYYAF2AL4AA7UNV74LBEC6COPCNEDAU7Z6JJGP6NZQRA4QGGSA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.273166902 (601h46m54.156580518s) [noise=true , address=77.90.31.18:4821 , piece_id=ZT34UIRYSJR7W63UBZP5433JLLRI3LX2D7VYLDEJCSFU26VG4TGQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.275724342 (601h46m54.15913796s) [noise=true , address=45.91.116.5:4323 , piece_id=MBT6BQWV46PCBWY46DJF5WGQRMECQEMLXZNUIOB6HQ3KFD7ZRKIA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278917091 (601h46m54.162330707s) [noise=true , address=45.91.116.12:4259 , piece_id=4HL24ORSSTX5HQ6CA5HKDUF7GDK72ELCZIV44TR4J247FR7SH4RQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.278945616 (601h46m54.162359243s) [noise=true , address=45.91.116.14:4222 , piece_id=MML4PDHMCEIHAZVDC2DBP45HNBGFL3GKANZKTCWRRQSOXFLVGH7A, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.283715439 (601h46m54.167129051s) [noise=true , address=45.91.116.6:4998 , piece_id=K7R34IO3TUEZJWXXKYXMJNP6TU6UA4KXTFC43XQIQCHDSM6NYROQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.284438587 (601h46m54.167852208s) [noise=true , address=79.127.219.48:4104 , piece_id=DFY23S6CJXIV2TOK3DM7IIMNKVA5K4OONUMDKO7RZI3NJMXPAWYQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.294188062 (601h46m54.1776017s) [noise=true , address=104.36.82.50:4025 , piece_id=K6X7WBLMFJ2OHOGZU5KHUETOW6JF6MJKE7XT2NZ5NYATISG6ULSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.434300533 (601h46m54.317714163s) [noise=true , address=45.91.95.3:4022 , piece_id=STAZHFFZ2HSXVHPYQSXBJ2G3U3E5CC6ICEWRNJQ25W7LPBU7TNZA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.435093696 (601h46m54.318507314s) [noise=true , address=45.91.95.4:4002 , piece_id=TRN4JO4MDWIL7NLFRBGC2TKCVY2UFTQW2BRAZHIYVAO7JMIFMF4Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.436583511 (601h46m54.31999714s) [noise=true , address=45.91.95.4:4008 , piece_id=2YB4HUDGAOGDY4X4KQIJWX6CMNU3DOZ327MEFTRG5BDZSGKY7XLQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.451259425 (601h46m54.334673202s) [noise=true , address=45.91.95.2:4028 , piece_id=WPSVDMJBZOXWJA6ZYTBYUKMHWQEVNWTUXZL3U4K5MAXQVHJW5SBQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518816025 (601h46m54.402229644s) [noise=true , address=79.127.201.215:4349 , piece_id=YUZGMQNUXBHPGVJAIWNGHGIXKZKQHEWXWIZYGSBP5W4235FJA2PA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: <nil> Jul 18 10:57:54.518860075 (601h46m54.402273706s) [seg_pos={0 0}, upload=481170]: Segment reached optimal threshold of 14 pieces. Jul 18 10:57:54.519304291 (601h46m54.402717909s) [noise=true , address=79.127.205.234:4617 , piece_id=EYKK7F2UJYXWJPR6T2KXUPWW6KBZ22YIZETLYDZPF5GSLXAFNICA, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUJcG8EBim135i6mRQQrwSBS5CFqVN5ipdFsGdWYH5Y9CxaUe): protocol: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49788->79.127.205.234:4617: use of closed network connection Jul 18 10:57:54.519319363 (601h46m54.402732994s) [noise=true , address=79.127.205.235:4685 , piece_id=PUFURTT4HVKI3VSZP3X33VD7ULRNLCNMXRMNKX4VZXF5ED5E6CSQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UK3CA8sureHVG8LghUPqYxYQgTY4gY7gaejKaMSBjbpY4kc1M): protocol: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53700->79.127.205.235:4685: use of closed network connection Jul 18 10:57:54.519384628 (601h46m54.402798246s) [noise=true , address=79.127.205.233:4647 , piece_id=NPEUREZXRRSGDPKJ5USSSRIOXEOYBYCRQVBAYNODPIZ7ABC47LWQ, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c19Vd3TVx59sX7RWYHvY99eMUQRqjJCyryCMZ5NdSdqeDQu9s): protocol: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46308->79.127.205.233:4647: use of closed network connection Jul 18 10:57:54.519402736 (601h46m54.402816368s) [noise=true , address=77.90.31.15:4704 , piece_id=NPGY5BTN46QRMZZXZHML3GZYGVBBFFTBSSCY4VGDCGGBKOSGFQ3Q, seg_pos={0 0}, upload=481170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: write tcp 139.59.144.22:58638->77.90.31.15:4704: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 10:57:54.519432414 (601h46m54.402846045s) [seg_pos={0 0}, upload=481170]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 18 10:57:54.519523054 (601h46m54.40293687s) [upload=481170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 10:57:54.519529627 (601h46m54.402943243s) [upload=481170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 10:57:54.662927129 (601h46m54.546340748s) [upload=481170]: Done uploading segments... Jul 18 10:57:54.662950246 (601h46m54.546363866s) [upload=481170]: Upload finished. err: <nil> Duration: 2.655549573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.....X.X..........X...............X..........X........] duration: 2.435654893s FAIL (2025-07-18 11:39:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== , upload=481608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 11:32:54.154838589 (602h21m54.038252206s) [upload=481608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 11:32:54.154843826 (602h21m54.038257453s) [upload=481608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 11:32:54.305725283 (602h21m54.189138898s) [upload=481608]: Done uploading segments... Jul 18 11:32:54.305745379 (602h21m54.189159007s) [upload=481608]: Upload finished. err: <nil> Jul 18 11:33:53.306124309 (602h22m53.189537928s) [upload=481619]: Starting upload Jul 18 11:33:53.306205552 (602h22m53.18961919s) [upload=481619]: Done starting upload Jul 18 11:33:53.306275544 (602h22m53.189689161s) [upload=481619]: Uploading segments... Jul 18 11:33:53.306475959 (602h22m53.189889586s) [upload=481619]: Got next segment. Inline: false Jul 18 11:33:53.306513703 (602h22m53.18992732s) [upload=481619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 11:33:53.306519968 (602h22m53.189933586s) [upload=481619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 11:33:53.419183467 (602h22m53.302597096s) [seg_pos={0 0}, upload=481619]: Begin upload segment... Jul 18 11:33:53.419316113 (602h22m53.302729751s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.419850102 (602h22m53.303263734s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420226364 (602h22m53.30364s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.420790778 (602h22m53.304204413s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421151603 (602h22m53.304565232s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421473878 (602h22m53.304887505s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.421830327 (602h22m53.305243957s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422004819 (602h22m53.305418454s) [seg_pos={0 0}, upload=481619]: Done begin upload segment. Jul 18 11:33:53.422067705 (602h22m53.305481341s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422255208 (602h22m53.305668836s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422510516 (602h22m53.305924152s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.422891686 (602h22m53.306305323s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423286438 (602h22m53.306700076s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.423650064 (602h22m53.307063701s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424001413 (602h22m53.30741505s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424340658 (602h22m53.307754287s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:53.424783800 (602h22m53.308197433s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Uploading piece... Jul 18 11:33:54.037250564 (602h22m53.920664177s) [upload=481619]: Next returned nil segment Jul 18 11:33:54.037268249 (602h22m53.920681862s) [upload=481619]: Waiting for error group managing segments... Jul 18 11:33:54.174223271 (602h22m54.057636898s) [noise=true , address=45.91.116.5:4299 , piece_id=6MPSMO2NGZF7KUKW62F6DV6MB7EUCUSBS37JTQWW4CFCFL6R5WUA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.175017283 (602h22m54.058430971s) [noise=true , address=45.91.116.13:4154 , piece_id=3DWHEX64WIUEFNTA7OQSI3N246VV5PL656S4BBZLKZRRFSYYPBSA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.186693354 (602h22m54.070106972s) [noise=true , address=77.90.31.10:4519 , piece_id=FZLX3JO2323KSQ2LRF7ZQIJ54SKUQOFADMJ4RV4MLDCR7LAKYLGA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194210810 (602h22m54.077624423s) [noise=true , address=45.91.116.12:4227 , piece_id=YZHWNLTXU4C7U6BD4ZMOASYHRKWOJX4SHTYSZCBFSCUCSIQGWKOA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.194454877 (602h22m54.077868504s) [noise=true , address=45.91.116.9:4261 , piece_id=RJPIMYLOKVTWT7OAPDSLZRDKAS6S24MDJJEDO7R6T6KES6QOTKJQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198427713 (602h22m54.08184133s) [noise=true , address=45.91.116.5:4298 , piece_id=MJCI34UOQBT7QVK5DJU35742ISYLWFUXYZKNZB2JDY2VVTVRNSKA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.198472131 (602h22m54.081885751s) [noise=true , address=45.91.116.8:4047 , piece_id=TH7446RXMI3N2KVGWO4RFEJRJT7VNQK3BCOV4LX3NPOKFUPDLPAQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.207083885 (602h22m54.090497512s) [noise=true , address=104.36.82.4:4801 , piece_id=DUDK4ODWMXKWOX3T44XQANH7K6IDJY5XUYAZAD6URYVQVD2EGGNA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.208816335 (602h22m54.09222996s) [noise=true , address=104.36.82.8:4971 , piece_id=GDLNL7U7CWXP32IIHOCDOCOFU5LFCQIOUE7Y5JR27BYGSGPBH5DQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:54.336523319 (602h22m54.219936957s) [noise=true , address=104.36.82.14:4223 , piece_id=HTLVOCDVQC25FPTK47EPLVESUQF4SQBARDGX34XF4H44S2KHR4KA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:33:55.287929075 (602h22m55.171342693s) [noise=true , address=45.91.116.10:4082 , piece_id=MTW24CAAS7GMDLOJPXNHI6GEAZMREH5EXQ4DSZGAOJFX6PMJRYLQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: <nil> Jul 18 11:34:23.031611818 (602h23m22.915025445s) [noise=true , address=77.90.31.18:4805 , piece_id=MEPMORRPK3YN2CH6I5CCXPWVKS2J2FLFWBZYXZPGPKTWLMVWO7UA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031747683 (602h23m22.915161316s) [noise=true , address=77.90.31.11:4555 , piece_id=36G5A5ZFW63MBSXUI5PUMXRWQWMIRUYB57WKV5MSFHIPOTWNGUQA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:43832->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031825318 (602h23m22.915238945s) [noise=true , address=77.90.31.16:4733 , piece_id=7GKK3MIAQZE55KUZ7CISUAKVOIY253FA5X3IPOEB46KIGZYHMQGQ, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:37494->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031906140 (602h23m22.915319768s) [noise=true , address=77.90.31.15:4699 , piece_id=YJBX34SLFMFNBBFJA3G2KDKB4HGIWY743XTQ5OHVFE5RJUDKYLXA, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:45528->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.031982671 (602h23m22.9153963s) [noise=true , address=77.90.31.14:4662 , piece_id=3QLWYPH4XRRGSGS2TWRLD33ZYGAWGQTUOHVRNARWKSFLAXDD5V6A, seg_pos={0 0}, upload=481619]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 139.59.144.22:44356->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 11:34:23.032005226 (602h23m22.915418863s) [seg_pos={0 0}, upload=481619]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 11:34:23.158376157 (602h23m23.041789785s) [upload=481619]: Done uploading segments... Jul 18 11:34:23.158427828 (602h23m23.041841454s) [upload=481619]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.435686667s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)