Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.738297576s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 01:09:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.738323192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.930680507s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 01:08:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.930708035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.78259309s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 01:07:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.782631044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 3.103904054s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 01:06:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.103934291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 3.128911748s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 01:06:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.128939912s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 3.020414009s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 01:05:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.020449758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.909801012s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 01:04:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.909825741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 3.038559071s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 01:04:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.038661992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.897258354s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 01:03:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.897294574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.704676495s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 01:03:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.704703693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 3.005942721s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 01:01:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.005982414s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.920656211s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 01:01:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.920715887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 3.015074081s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:59:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.0151003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 3.152146091s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:59:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.152168953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 3.067016168s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:58:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.067044683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 3.174697177s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:57:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.174734208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.095371941s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:56:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.09540021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.846082903s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:55:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.846140282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 3.075362678s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:53:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.075387348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 3.083400037s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:53:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.083432139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.871117345s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:52:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.871140681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 29.573365252s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:51:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 29.573419123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 3.176537082s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:50:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.176569627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 3.083235901s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:50:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.083261471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 3.060666868s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:49:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.060753485s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 3.150026665s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:49:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.150049849s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.958349614s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:47:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.958373252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.867056212s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:46:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.867082232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 3.2096599s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:45:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.209684474s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.632639192s trace: https://jaeger.dev.storj.io/trace/23bfa65069eba8a5 FAIL (2026-03-12 00:45:38 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.632662695s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.677490631s trace: https://jaeger.dev.storj.io/trace/23bfa65069eba8a5 FAIL (2026-03-12 00:45:23 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.677509306s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 3.162136571s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:45:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.162160551s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 2.118355606s trace: https://jaeger.dev.storj.io/trace/23bfa65069eba8a5 FAIL (2026-03-12 00:44:33 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.118377332s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 1.665033044s trace: https://jaeger.dev.storj.io/trace/23bfa65069eba8a5 FAIL (2026-03-12 00:44:01 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.665112433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 3.287051741s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:43:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.287089912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 2.098001178s trace: https://jaeger.dev.storj.io/trace/23bfa65069eba8a5 FAIL (2026-03-12 00:43:33 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.098025331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.974346824s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:43:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.974377272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 3.205612379s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:42:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.205685072s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 2.148797502s trace: https://jaeger.dev.storj.io/trace/23bfa65069eba8a5 FAIL (2026-03-12 00:42:20 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.148820275s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.68786861s trace: https://jaeger.dev.storj.io/trace/23bfa65069eba8a5 FAIL (2026-03-12 00:41:30 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.687898623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 3.122593839s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:41:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.122625688s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.635065633s trace: https://jaeger.dev.storj.io/trace/23bfa65069eba8a5 FAIL (2026-03-12 00:41:06 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.635098532s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.595010305s trace: https://jaeger.dev.storj.io/trace/23bfa65069eba8a5 FAIL (2026-03-12 00:40:33 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.595037442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 3.30999317s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:40:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.31002675s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 1.56619978s trace: https://jaeger.dev.storj.io/trace/23bfa65069eba8a5 FAIL (2026-03-12 00:39:49 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.566229632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 3.307672468s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:39:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.307717587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 3.133863795s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:39:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.133898827s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 2.131089536s trace: https://jaeger.dev.storj.io/trace/23bfa65069eba8a5 FAIL (2026-03-12 00:39:01 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.131117446s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 1.550490303s trace: https://jaeger.dev.storj.io/trace/23bfa65069eba8a5 FAIL (2026-03-12 00:38:25 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.550517434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 3.242717046s trace: https://jaeger.dev.storj.io/trace/174b7068acd9c74d FAIL (2026-03-12 00:38:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10837 (32h20m37.501980439s) [upload=31037]: Uploading segments... Mar 12 00:31:34.486776050 (32h20m37.502445657s) [upload=31037]: Got next segment. Inline: false Mar 12 00:31:34.486855931 (32h20m37.502525535s) [upload=31037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 12 00:31:34.486862930 (32h20m37.502532538s) [upload=31037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 12 00:31:34.602695736 (32h20m37.618365349s) [seg_pos={0 0}, upload=31037]: Begin upload segment... Mar 12 00:31:34.602763613 (32h20m37.618433214s) [seg_pos={0 0}, upload=31037]: Done begin upload segment. Mar 12 00:31:34.602822074 (32h20m37.618491684s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.602842924 (32h20m37.618512523s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.603681917 (32h20m37.619351527s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.604424128 (32h20m37.620093734s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605011842 (32h20m37.620681449s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.605528765 (32h20m37.621198373s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606168019 (32h20m37.621837625s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.606859614 (32h20m37.622529223s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607454731 (32h20m37.623124341s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.607930410 (32h20m37.623600012s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.609131954 (32h20m37.624801562s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610330433 (32h20m37.626000044s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.610902935 (32h20m37.626572544s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.612789464 (32h20m37.628459074s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.613671297 (32h20m37.629340914s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.615326520 (32h20m37.630996128s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.618326641 (32h20m37.633996248s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:34.619644361 (32h20m37.635313972s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Uploading piece... Mar 12 00:31:35.179232447 (32h20m38.194902033s) [upload=31037]: Next returned nil segment Mar 12 00:31:35.179255069 (32h20m38.194924673s) [upload=31037]: Waiting for error group managing segments... Mar 12 00:31:35.324345351 (32h20m38.34001496s) [noise=true , address=145.79.227.11:4748 , piece_id=ANRQUH4Q6U76CKX5IJNPB4DT5ZWY2Z6DL7M445CTAMCZY6TPEBIQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.329514593 (32h20m38.345184211s) [noise=true , address=145.79.227.17:4984 , piece_id=6OY4DUI4GNEYOL4YGT2DW7HCZJ5LAA73COBG7BOQ3SDBNJIJN4KA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.332335907 (32h20m38.348005511s) [noise=true , address=45.91.116.10:4101 , piece_id=4CLKQAD47CVND7FENO6QCNB6TSJWDL7IKPVZKB5UC23GQH4V7HAA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.347415595 (32h20m38.363085195s) [noise=true , address=145.79.227.16:4928 , piece_id=374MLV3TYE526GE3SA4H4C5VASLIR5F3CGVR26BL6UYXI2JEX2RA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.683227566 (32h20m38.698897167s) [noise=true , address=167.17.72.69:4724 , piece_id=NICLUZ72ZUDOVJKWCPBDK4ZSSKGZVJ3Y3MAZL2ICCCCOMDM3734Q, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.689845167 (32h20m38.705514768s) [noise=true , address=167.17.72.71:4809 , piece_id=2VMRO3MCIVCWI5DRUONAT7XHVNJWMKP6UCVPFNXF3CQPGP2D5SFQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.698339628 (32h20m38.714009233s) [noise=true , address=167.17.72.68:4657 , piece_id=ZO4UJRKHIFS5IHSQ6PKTZ6AJLLG6BC3DQT5HC3OIO6UPWNP7B43A, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.838364650 (32h20m38.854034259s) [noise=true , address=193.149.164.68:4033 , piece_id=ZHFB4E72IZ4SDFOXKXKIFVBYWZ6L52WSASVEV55LL5GOCXS3I3DA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.859483214 (32h20m38.87515283s) [noise=true , address=64.49.13.138:4482 , piece_id=AS7MWR7GBLRJNHSDGY5OVWCY77IKVW2SVHFTVBZSQUCWRQZMBIZA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.860550874 (32h20m38.876220478s) [noise=true , address=193.149.164.70:4118 , piece_id=34MVMVHDGATXPQIQZ2WVBUJYP5NS35TWOMBU7X5LAUNXRZRXOLHQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.870706156 (32h20m38.886375746s) [noise=true , address=193.149.164.66:4977 , piece_id=VXZEPOAY4KXDW3BUCFHWQT7PAEU5WL2BOYF7KUYRI2V72NQVWAFA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:35.894591310 (32h20m38.910260911s) [noise=true , address=193.149.164.71:4291 , piece_id=EWQTPEA7C6BSMOJJZVFUE3TBF7WF2SHHAKOL3QKO4CPBZPWL35BA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:31:37.536870602 (32h20m40.552540214s) [noise=true , address=79.127.205.237:4236 , piece_id=3BU6NGWK2B4S6H5HXJSMSM3ZJEDZZFMEPTADQXJLOAZ2G5EYHOUQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: <nil> Mar 12 00:32:34.202030668 (32h21m37.217700274s) [noise=true , address=77.90.31.14:4685 , piece_id=DVFFETGMI7KCULN5L7EWGCT7SFHAXPJ7NH2NEDO6O3LBYYCUEOZQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: write tcp 139.59.144.22:44908->77.90.31.14:4685: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202352883 (32h21m37.218022493s) [noise=true , address=77.90.31.16:4750 , piece_id=TENSMPQBKXM737MG6RMKQJ2JWZF7UFBEAD7VE27DEJ7KR3ZLXDRQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202537306 (32h21m37.218206917s) [noise=true , address=77.90.31.19:4846 , piece_id=JY6E6P6FTE7H5DLV5F66AILET4BOAQ37677GPJRARJSHXKZBWVCQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:41948->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202626126 (32h21m37.218295733s) [noise=true , address=77.90.31.21:4895 , piece_id=VRQUXW53SGSXBNQRLWZNV3GJWM655MKVWDCV4XUK3GW34CX75PVQ, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.202960912 (32h21m37.218630519s) [noise=true , address=77.90.31.18:4826 , piece_id=U6ZYNCYOHPC4UDLETIDEDOICOT3MAUVNBNK3VXADV2N5Y4PFSXNA, seg_pos={0 0}, upload=31037]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: rpc: dial tcp 77.90.31.18:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 12 00:32:34.203006347 (32h21m37.218675939s) [seg_pos={0 0}, upload=31037]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 12 00:32:34.306147527 (32h21m37.321817139s) [upload=31037]: Done uploading segments... Mar 12 00:32:34.306191442 (32h21m37.321861053s) [upload=31037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.242733007s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)