Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.270018199s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:05:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.270047244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.275978597s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:04:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.276005755s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.265166834s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:04:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.265191011s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.299529134s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:03:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.299614924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.342046319s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:02:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.342073578s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.358939757s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:02:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.358977943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.401752691s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:01:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.401793209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.54616909s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:00:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.546286417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.355756045s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:59:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.355782817s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.407643258s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:58:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.407666558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.51332008s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:58:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.513421205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.746496297s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:57:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.746532164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.344657753s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:56:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.344690742s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.391639366s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:55:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.391672713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.391504198s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:54:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.391598572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.442988388s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:53:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.4430803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.530288463s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:53:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.53031199s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.332980046s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:52:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.333010834s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.715015657s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:51:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.715043016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.447880829s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:50:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.447907576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.444641005s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:49:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.44466962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.375460788s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:48:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.375561013s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.283087063s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:47:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.283111737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.538916775s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:47:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.539028075s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.326001812s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:46:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.326034498s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.27521075s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:45:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.27523136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.313331176s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:44:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.313354601s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.341298413s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:43:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.341333618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.47015s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:42:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.470175943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.37880345s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:42:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.378829484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.475049381s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:41:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.475077334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.378248121s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:41:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.378280488s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.39962394s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:40:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.399735645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.110742327s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 15:40:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.110762869s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.691440896s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:51:31 UTC) orbiter warning: context deadline exceeded Duration: 2.69146753s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.762202151s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:50:44 UTC) orbiter warning: context deadline exceeded Duration: 2.762228651s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.601170042s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:49:55 UTC) orbiter warning: context deadline exceeded Duration: 2.601194768s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.605024475s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:48:47 UTC) orbiter warning: context deadline exceeded Duration: 2.605046366s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.492094727s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:48:22 UTC) orbiter warning: context deadline exceeded Duration: 2.492121158s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.640447655s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:47:17 UTC) orbiter warning: context deadline exceeded Duration: 2.640467422s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.625838817s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:46:19 UTC) orbiter warning: context deadline exceeded Duration: 2.625860818s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.61988211s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:45:46 UTC) orbiter warning: context deadline exceeded Duration: 2.619905574s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.623318939s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:45:29 UTC) orbiter warning: context deadline exceeded Duration: 2.623339416s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.728771687s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:44:46 UTC) orbiter warning: context deadline exceeded Duration: 2.728861219s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.801128624s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:44:12 UTC) orbiter warning: context deadline exceeded Duration: 2.801152443s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.894515543s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:43:28 UTC) orbiter warning: context deadline exceeded Duration: 2.89455113s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.565223789s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:42:43 UTC) orbiter warning: context deadline exceeded Duration: 2.565268122s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.797343303s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:41:31 UTC) orbiter warning: context deadline exceeded Duration: 2.797373557s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.899460397s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:41:07 UTC) orbiter warning: context deadline exceeded Duration: 2.899488573s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.730004047s trace: https://jaeger.dev.storj.io/trace/628fb9df9c6707de FAIL (2025-12-01 23:40:14 UTC) orbiter warning: context deadline exceeded Duration: 2.730037933s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)