Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.314460977s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:15:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.314498832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.335025514s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:14:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.335053249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.305693513s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:13:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.30572474s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.420778438s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:13:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.420805342s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.421513573s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:12:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.421614785s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.469160049s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:11:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.469189704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.353940747s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:10:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.353969344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.324907797s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:09:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.324937009s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.415041077s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:08:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.415069421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.40293941s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:08:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.402979637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.49135701s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:06:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.491390201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.338337205s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:06:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.338395038s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.466482739s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:06:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.466512698s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.378725497s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:04:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.378750956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.408975303s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:04:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.409004747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.527601975s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:03:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.527633489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.313480015s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:03:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.313509862s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.465425336s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:02:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.465457361s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.421660596s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:01:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.421688796s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.362339977s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 07:00:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.36236902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.362004158s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:59:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.362034054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.386329995s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:59:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.386363364s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.368858917s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:58:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.368887553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.36970804s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:57:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.369740102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.293963955s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:56:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.29398367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.264970079s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:56:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.265084902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.306640272s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:55:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.306669417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.343811039s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:55:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.343860725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.393308168s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:54:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.393356468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.410816049s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:53:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.410842634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.364576128s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:52:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.36462387s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.258660564s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:52:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.258685354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.444821906s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:51:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.444868269s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.390268948s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:50:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.390299321s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.441451452s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:49:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.44148783s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.376930611s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:49:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.376958469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 19.585572399s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:48:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 19.585594549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.109760201s trace: https://jaeger.dev.storj.io/trace/5403b15b90934969 FAIL (2025-12-22 06:47:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 06:47:13.390879689 (399h18m53.371232325s) [upload=351224]: Starting upload Dec 22 06:47:13.390933946 (399h18m53.371286588s) [upload=351224]: Done starting upload Dec 22 06:47:13.391022385 (399h18m53.371375007s) [upload=351224]: Uploading segments... Dec 22 06:47:13.391257160 (399h18m53.371609787s) [upload=351224]: Got next segment. Inline: false Dec 22 06:47:13.391304735 (399h18m53.371657378s) [upload=351224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 06:47:13.391309842 (399h18m53.371662478s) [upload=351224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 06:47:13.503578512 (399h18m53.48393115s) [seg_pos={0 0}, upload=351224]: Begin upload segment... Dec 22 06:47:13.503670819 (399h18m53.484023465s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504640710 (399h18m53.484993414s) [seg_pos={0 0}, upload=351224]: Done begin upload segment. Dec 22 06:47:13.504785964 (399h18m53.485138878s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.504916823 (399h18m53.485269465s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505227903 (399h18m53.485580539s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505250192 (399h18m53.485602828s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505519011 (399h18m53.485871647s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505579890 (399h18m53.485932526s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.505839462 (399h18m53.486192109s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506000955 (399h18m53.486353591s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506194887 (399h18m53.486547527s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506408314 (399h18m53.486760957s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506480917 (399h18m53.486833557s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506749655 (399h18m53.48710229s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.506759456 (399h18m53.487112092s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.507955888 (399h18m53.488308525s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:13.509580758 (399h18m53.489933403s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Uploading piece... Dec 22 06:47:14.116474867 (399h18m54.096827503s) [upload=351224]: Next returned nil segment Dec 22 06:47:14.116487753 (399h18m54.096840382s) [upload=351224]: Waiting for error group managing segments... Dec 22 06:47:14.252424715 (399h18m54.232777352s) [noise=true , address=77.90.31.19:4842 , piece_id=EM2LSIEISDYOD4M6GEBXZ3FS3TFSFTPNVBGWL63GY3KNSRNDB7LQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.259199346 (399h18m54.239551983s) [noise=true , address=45.91.116.6:4976 , piece_id=U4OR7ZIHIU5GSFU7V2CI4D4BVBNHFK6IXQZPEJXO5EZYFU2Q5HSA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.263033341 (399h18m54.243385978s) [noise=true , address=145.79.227.16:4910 , piece_id=XVCKNOPRCVY5DHZ4O65RJCEWHOAGFO7QNUL5NQX67FCAXFWGFFBA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.268392055 (399h18m54.2487447s) [noise=true , address=145.79.227.45:4230 , piece_id=XCWRUKHSKBWUNMQ7PAGAZCUGU2JGYHR3YN7A73VXSSZGH4SRHDXQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272217620 (399h18m54.252570256s) [noise=true , address=145.79.227.34:4905 , piece_id=FMMFST4XFUFCC45LJASZWNRU2GKJOLYFFBDXZCSHPULN7PYRA34Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.272621367 (399h18m54.252974009s) [noise=true , address=145.79.227.4:4187 , piece_id=7FXNQSCCEXKMRNQ4TMIETZMXSL34LGIHZW43RRQUYLBVC6JC2RRA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.274087214 (399h18m54.254439851s) [noise=true , address=77.90.31.14:4662 , piece_id=QHEQJTJU43V4QJSUJH5UQRUNO7BIG5TGVMFVUZEAAGSCTV4LJGOQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.281307529 (399h18m54.261660168s) [noise=true , address=45.91.116.5:4298 , piece_id=Z3EZHUWSEDLLQO45UJQWEGDI2T2EGFIN5QK44KEJKMFHJS47YKGQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.290977855 (399h18m54.2713305s) [noise=true , address=145.79.227.51:4410 , piece_id=CFOA5CQ4SQYWJB64G22GSFUU42E2PTGYSSVM53WXFUHOUVP6J47A, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:14.305725196 (399h18m54.28607784s) [noise=true , address=145.79.227.41:4086 , piece_id=ADAWH4SVXEJ5HVMLBN7QXLN5CTJ4XYCUSDTEMBDTRPBMJYO47SIQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:21.393190149 (399h19m1.373542789s) [noise=true , address=145.79.227.23:4547 , piece_id=WPZ3MCPYCISSFJFCU5ROE2QG6NIDAMJKL2XXCPKD6VFKCNLMC7LA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:22.349672394 (399h19m2.33002503s) [noise=true , address=145.79.227.28:4727 , piece_id=77AWXI32H7JJCVHHC23OXHYURQYJUN7EDEOZGYI2TBKGBBLHJCEA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: <nil> Dec 22 06:47:43.106830367 (399h19m23.087183004s) [noise=true , address=145.79.227.29:4942 , piece_id=GSFL36URKGAC7IFVCLJW4EVRLUONYQR6OAFHSR6LCCT4KGG74QFA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107030968 (399h19m23.087383607s) [noise=true , address=145.79.227.40:4049 , piece_id=JDNXRRGRNTGSM4WN3IZTBTLVI3C5LMIXICTEVE6B2DOLEXY2JG2Q, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107221616 (399h19m23.087574252s) [noise=true , address=145.79.227.8:4329 , piece_id=CIDYWZXFMW6QVG7F3W36JOIVFL6L5WA4VPYHKZN7BZ2AO4FPCHCQ, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107779626 (399h19m23.088132267s) [noise=true , address=145.79.227.32:4835 , piece_id=FSFZYWEJHZZSWVCEEFGNXPOVLV2HNWOVJKV3DGPQ36PLIF3P2MWA, seg_pos={0 0}, upload=351224]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 06:47:43.107809812 (399h19m23.088162439s) [seg_pos={0 0}, upload=351224]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 06:47:43.213405603 (399h19m23.193758247s) [upload=351224]: Done uploading segments... Dec 22 06:47:43.213448935 (399h19m23.193801581s) [upload=351224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.109773996s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.587628855s trace: https://jaeger.dev.storj.io/trace/37924a1da3fa03fb FAIL (2025-12-22 04:52:57 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 185.244.226.5:443: i/o timeout Duration: 2.58765861s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 111.430826ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/4862db1d0e6a8f0f FAIL (2025-12-22 04:52:35 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 185.244.226.3:443: i/o timeout Duration: 111.451038ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.781712083s trace: https://jaeger.dev.storj.io/trace/37924a1da3fa03fb FAIL (2025-12-22 04:52:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 185.244.226.5:443: i/o timeout Duration: 2.78173826s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 105.861308ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/4862db1d0e6a8f0f FAIL (2025-12-22 04:51:53 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 185.244.226.3:443: i/o timeout Duration: 105.893899ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.172153896s trace: https://jaeger.dev.storj.io/trace/37924a1da3fa03fb FAIL (2025-12-22 04:51:10 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 185.244.226.5:443: i/o timeout Duration: 3.172183907s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 116.420967ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/4862db1d0e6a8f0f FAIL (2025-12-22 04:50:59 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 185.244.226.3:443: i/o timeout Duration: 116.441305ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 122.310535ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/4862db1d0e6a8f0f FAIL (2025-12-22 04:50:22 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 185.244.226.3:443: i/o timeout Duration: 122.352392ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.638950153s trace: https://jaeger.dev.storj.io/trace/37924a1da3fa03fb FAIL (2025-12-22 04:50:17 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 185.244.226.5:443: i/o timeout Duration: 2.638992423s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 3.022268715s trace: https://jaeger.dev.storj.io/trace/37924a1da3fa03fb FAIL (2025-12-22 04:49:32 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 185.244.226.5:443: i/o timeout Duration: 3.022300689s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 111.257699ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/4862db1d0e6a8f0f FAIL (2025-12-22 04:49:16 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 185.244.226.3:443: i/o timeout Duration: 111.2957ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.942653907s trace: https://jaeger.dev.storj.io/trace/37924a1da3fa03fb FAIL (2025-12-22 04:49:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 185.244.226.5:443: i/o timeout Duration: 2.942697727s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 112.17242ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/4862db1d0e6a8f0f FAIL (2025-12-22 04:48:57 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 185.244.226.3:443: i/o timeout Duration: 112.223648ms edge gateway vhost bgp us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)