Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.389206463s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:51:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.389225891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.42469702s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:51:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.424732546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.456751674s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:50:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.456777445s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.727899831s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:49:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.72793627s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.430319151s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:48:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.430346227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.261500663s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:48:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.261529243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.296410761s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:47:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.296459384s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.35445799s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:46:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.354496468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.483058524s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:46:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.483087363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.541364572s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:45:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.541394646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.358651314s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:45:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.358682459s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.34609284s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:44:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.346118096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.289157637s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:43:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.289247667s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.405720898s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:42:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.405765873s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.381884403s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:41:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.381915631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.459130214s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:40:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.459156731s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.400709137s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:40:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.400738078s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 4.447046545s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:39:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.447077464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.452602714s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:38:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.452622871s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.110483239s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:38:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.110509614s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.627131996s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:43:25 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.627182581s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.601338815s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:42:45 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.602234406s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.659827189s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:42:25 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.659847757s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 1.728507007s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:41:29 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.728527569s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 1.732505436s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:40:16 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.732537287s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 1.600030133s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:39:32 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.600067868s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.599712752s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:38:56 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.599759755s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.616849691s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:38:37 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.61687385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.696367547s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:37:35 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.69640921s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 2.138592876s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:36:38 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.138617759s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 2.168544264s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:35:30 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.168578851s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 2.144741013s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:35:10 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.144763228s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 1.67499529s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:34:04 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.675109853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 1.705374111s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:33:21 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.705457581s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 1.617347535s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:32:15 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.617373002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 1.668242012s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:31:42 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.668265019s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 1.86828183s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:30:57 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.868466255s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 1.705048938s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:29:58 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.70507272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 1.684714486s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:29:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.684814849s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 1.850643607s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:29:07 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.850669386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 1.610623189s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:28:10 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.61065489s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 1.712119589s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:27:51 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.712147069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 1.525175874s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:27:18 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.525198481s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 2.3333866s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:26:36 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.333440224s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 2.148021083s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:25:26 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.148047893s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 1.56191162s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:24:27 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.561938153s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 1.663785558s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:23:24 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.663818619s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 1.707235634s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:22:45 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.70726018s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 1.536467591s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:22:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.536498837s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 2.114653423s trace: https://jaeger.dev.storj.io/trace/31d68db84cbfd08e FAIL (2025-12-09 08:21:22 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.114753452s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)