Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.496304438s FAIL (2025-09-04 16:17:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.49633607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.545792276s FAIL (2025-09-04 16:16:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.545829741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.420327243s FAIL (2025-09-04 16:15:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.420445878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.546279634s FAIL (2025-09-04 16:14:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.546417954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.453473717s FAIL (2025-09-04 16:14:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.453499829s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.454725474s FAIL (2025-09-04 16:13:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.454756989s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.381450465s FAIL (2025-09-04 16:12:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.381481302s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.551949515s FAIL (2025-09-04 16:12:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.551985693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.562961349s FAIL (2025-09-04 16:12:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.562992754s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.4332574s FAIL (2025-09-04 16:11:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.433296228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.579898514s FAIL (2025-09-04 16:10:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.57994349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.570045557s FAIL (2025-09-04 16:09:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.570083482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.375929992s FAIL (2025-09-04 16:09:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.375967488s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.600058499s FAIL (2025-09-04 16:08:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.600282374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.385957919s FAIL (2025-09-04 16:07:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.385996613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.879503131s FAIL (2025-09-04 16:07:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.879558134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.217946494s FAIL (2025-09-04 16:05:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.218075949s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.431887022s FAIL (2025-09-04 16:05:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.431927968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.370955634s FAIL (2025-09-04 16:04:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.370988516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.607068323s FAIL (2025-09-04 16:03:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.607102336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.12830384s FAIL (2025-09-04 16:03:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 4 16:02:42.909719129 (145h55m3.402151563s) [upload=128456]: Starting upload Sep 4 16:02:42.909796536 (145h55m3.402228978s) [upload=128456]: Done starting upload Sep 4 16:02:42.909970029 (145h55m3.402402466s) [upload=128456]: Uploading segments... Sep 4 16:02:42.910279905 (145h55m3.402712347s) [upload=128456]: Got next segment. Inline: false Sep 4 16:02:42.910327706 (145h55m3.402760153s) [upload=128456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 4 16:02:42.910334234 (145h55m3.402766684s) [upload=128456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 4 16:02:43.030362651 (145h55m3.522795101s) [seg_pos={0 0}, upload=128456]: Begin upload segment... Sep 4 16:02:43.030428427 (145h55m3.522860873s) [seg_pos={0 0}, upload=128456]: Done begin upload segment. Sep 4 16:02:43.030481197 (145h55m3.522913646s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030491260 (145h55m3.522923712s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.030637146 (145h55m3.523069594s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031039875 (145h55m3.523472328s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031052847 (145h55m3.523485298s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031473422 (145h55m3.523905873s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031487613 (145h55m3.523920067s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.031942053 (145h55m3.524374506s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032440371 (145h55m3.524872823s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.032857734 (145h55m3.525290184s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033263504 (145h55m3.525695956s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.033651092 (145h55m3.526083522s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034034716 (145h55m3.526467276s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034404674 (145h55m3.526837127s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.034763408 (145h55m3.527195842s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.042130641 (145h55m3.53456307s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Uploading piece... Sep 4 16:02:43.142071932 (145h55m3.634504383s) [noise=true , address=77.90.31.16:4733 , piece_id=YRXPBCNEYLXLHHTAQ5CICHWNQ53VOOGQRHJURRGHTKM66SFMXALQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 139.59.144.22:45412->77.90.31.16:4733: write: connection refused; conn closed: op Sep 4 16:02:43.649162898 (145h55m4.14159534s) [upload=128456]: Next returned nil segment Sep 4 16:02:43.649201552 (145h55m4.141633994s) [upload=128456]: Waiting for error group managing segments... Sep 4 16:02:43.787153228 (145h55m4.279585661s) [noise=true , address=45.91.116.12:4226 , piece_id=GCDHGJXTWPEYH5ZA4LUZX35JIYT7SGNZUWNIQNNKONU67UT6MYVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.794033433 (145h55m4.286465869s) [noise=true , address=77.90.31.14:4662 , piece_id=Q7HU2WZ2C2QWQWHFR74FRNZQ67JECTP67JVCTV5VYJFR7NNBEMVA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.797370363 (145h55m4.289802805s) [noise=true , address=45.91.116.5:4297 , piece_id=2XX3YAVK2QQBKVAX4ACNAPRHOKFG5IZMPDGJNUX6REULDZQYVWUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.800401208 (145h55m4.292833644s) [noise=true , address=45.91.116.7:4011 , piece_id=RFKP3SAEJHQKFY7WTYEXY2ZTDM62DIZ56EAFG6EFSAI5KHOA66LA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.802148843 (145h55m4.294581287s) [noise=true , address=45.91.116.5:4299 , piece_id=G7I2KD76ZF67MIQ4Q64SMBLALPTH5OPMJ4DXLH2W4RJXK3XJLTZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.806652318 (145h55m4.299084746s) [noise=true , address=45.91.116.11:4118 , piece_id=4QJJBZX7GMEJUFOOOHEVC7I5USDDU7XWRT4GVFJGJJHZNQGX444A, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.809158593 (145h55m4.301591024s) [noise=true , address=45.91.116.13:4153 , piece_id=K5XSNOYMSMKYGGF2BBQVNFGPTKECW3UWLLMTEM3K3G4GEWBWCGQA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.815930422 (145h55m4.308362866s) [noise=true , address=45.91.116.4:4938 , piece_id=OQEE2FHNCWZBD6LRZL7VWLLTBXN5LSQYDNE2OX57Z4UUI5CIHHHA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.816434663 (145h55m4.308867097s) [noise=true , address=45.91.116.4:4939 , piece_id=YZ5NE5WC5SVLEJG4KVKENJPZODUECGKM7CQ6BYU3B6OBZV7IK4RA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.823376139 (145h55m4.315808574s) [noise=true , address=45.91.116.12:4225 , piece_id=6EJGEBNHULNADMZ5NVBXWYNENHPSMZCTCMB2Y2W2OEYYQYOGNZUQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.828695794 (145h55m4.321128229s) [noise=true , address=45.91.116.11:4117 , piece_id=YFZOLJMOVQUL6JWW4XQTSRUW7BJJS73IIUZ2ZLIVG4A3KS3ER7XQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:43.837433498 (145h55m4.329865935s) [noise=true , address=77.90.31.12:4592 , piece_id=LL2QYC5K6DMLRBVVNFSA4SEFVEI2P3TKDDAN75FJKKITJ6SR2S6Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: <nil> Sep 4 16:02:55.067986508 (145h55m15.560418954s) [noise=true , address=77.90.31.15:4697 , piece_id=4DJYLDGDAMZGBROAAIMAFCYAEFXY7Q5HM2QHSTYALGB4T2HRGIZA, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:51706->77.90.31.15:4697: read: connection reset by peer; conn closed: op; piecestore: piecestore close: conn closed: op Sep 4 16:03:12.597282068 (145h55m33.0897145s) [noise=true , address=77.90.31.19:4842 , piece_id=NAY24XIAA4QPF5XVCIXRXXABLQYSQJZXDBKGBCJLEIG32KGHAS2Q, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597421723 (145h55m33.089854166s) [noise=true , address=77.90.31.19:4841 , piece_id=6ES56EBISG2F5LBC6CTGL5DO4XG4YJ566EJMJP3YTK2POM7BNWGQ, seg_pos={0 0}, upload=128456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Sep 4 16:03:12.597445781 (145h55m33.089878226s) [seg_pos={0 0}, upload=128456]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 4 16:03:12.723001789 (145h55m33.215434225s) [upload=128456]: Done uploading segments... Sep 4 16:03:12.723050678 (145h55m33.215483121s) [upload=128456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.128333717s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.106584917s FAIL (2025-09-03 21:32:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.106625247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.207452895s FAIL (2025-09-03 21:31:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.207485489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.865356775s FAIL (2025-09-03 21:30:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.865386029s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.712465677s FAIL (2025-09-03 21:30:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.712493688s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.881383842s FAIL (2025-09-03 21:29:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.881486527s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.48623589s FAIL (2025-09-03 21:29:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.486279521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.382492247s FAIL (2025-09-03 21:29:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.382530693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.398072149s FAIL (2025-09-03 21:29:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.398103378s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 3.453247562s FAIL (2025-09-03 21:29:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.453273971s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.508141313s FAIL (2025-09-03 21:28:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.508174329s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 2.904259627s FAIL (2025-09-03 21:27:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.904338919s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X......X....................................................] duration: 2.34981479s FAIL (2025-09-03 21:27:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.349850416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 2.736940152s FAIL (2025-09-03 21:26:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.736967286s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 2.786898043s FAIL (2025-09-03 21:26:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.786954848s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX......X...................................................] duration: 2.621742703s FAIL (2025-09-03 21:26:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.621774352s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 2.912147601s FAIL (2025-09-03 21:25:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.912171404s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XX......X..................................................] duration: 2.505279139s FAIL (2025-09-03 21:25:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.50530924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..XX......X.................................................] duration: 2.395666062s FAIL (2025-09-03 21:25:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.395708961s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 2.758259342s FAIL (2025-09-03 21:24:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.758289067s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...XX......X................................................] duration: 2.7739966s FAIL (2025-09-03 21:24:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.774031132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....XX......X...............................................] duration: 2.643190662s FAIL (2025-09-03 21:24:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.643318966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....XX......X..............................................] duration: 3.057456828s FAIL (2025-09-03 21:23:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.057490068s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 2.823799348s FAIL (2025-09-03 21:23:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.824222971s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 2.850943932s FAIL (2025-09-03 21:22:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.850963319s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.....XX......X.............................................] duration: 2.410451332s FAIL (2025-09-03 21:22:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.41048181s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 2.839238234s FAIL (2025-09-03 21:22:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.839306201s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.....XX......X............................................] duration: 2.942921559s FAIL (2025-09-03 21:21:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.942946482s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 2.757931617s FAIL (2025-09-03 21:21:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.757951506s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 3.183006103s FAIL (2025-09-03 21:21:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.183034383s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)