Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 1.645240574s trace: https://jaeger.dev.storj.io/trace/4b29fbcd50a468d1 FAIL (2025-12-10 04:54:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.645263851s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 2.090286563s trace: https://jaeger.dev.storj.io/trace/4b29fbcd50a468d1 FAIL (2025-12-10 04:54:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.090310508s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 1.599051299s trace: https://jaeger.dev.storj.io/trace/4b29fbcd50a468d1 FAIL (2025-12-10 04:53:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.599075453s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 1.6225558s trace: https://jaeger.dev.storj.io/trace/4b29fbcd50a468d1 FAIL (2025-12-10 04:53:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.623250083s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 1.742162572s trace: https://jaeger.dev.storj.io/trace/4b29fbcd50a468d1 FAIL (2025-12-10 04:52:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.742216493s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 2.242594895s trace: https://jaeger.dev.storj.io/trace/4b29fbcd50a468d1 FAIL (2025-12-10 04:51:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.242634911s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 2.174609247s trace: https://jaeger.dev.storj.io/trace/4b29fbcd50a468d1 FAIL (2025-12-10 04:50:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.174731656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 1.597381078s trace: https://jaeger.dev.storj.io/trace/4b29fbcd50a468d1 FAIL (2025-12-10 04:49:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.597410843s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 1.584671121s trace: https://jaeger.dev.storj.io/trace/4b29fbcd50a468d1 FAIL (2025-12-10 04:48:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.584693851s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001846383s trace: https://jaeger.dev.storj.io/trace/4b29fbcd50a468d1 FAIL (2025-12-10 04:48:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001866784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.432779551s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:24:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.432803926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.298892271s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:22:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.298985943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.506681606s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:22:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.506705252s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.39700697s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:21:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.397052143s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.395201279s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:20:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.395220074s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.387233258s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:19:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.387255992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.428709593s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:18:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.428753906s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.40112286s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:17:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.401151776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.335497942s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:16:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.335527747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.593790235s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:15:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.593897687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.288415958s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:14:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.288444998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.528205916s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:13:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.528235193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.434380108s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:12:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.434423456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.413008585s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:11:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.413044045s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.432724401s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:11:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.432753012s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.445585496s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:10:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.445674142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.479102946s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:09:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.479129237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.297400119s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:09:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.297421242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.411389657s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:08:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.411419553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.370702949s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:07:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.370746095s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.417439091s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:06:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.417463917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.389369017s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:05:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.38940618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.389392092s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:05:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.389433473s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.408661403s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:04:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.408715051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.434466784s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:03:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.434489843s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.423070186s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:03:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.423202711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.243673231s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:02:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.243719691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.267513642s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:01:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.267561167s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.376806647s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 13:00:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.376831778s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.308337258s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:59:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.308364628s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.403708713s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:59:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.403736632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.464511192s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:58:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.464534066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.407243328s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:58:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.4072916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.426184195s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:57:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.42621139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.382348904s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:56:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.382374371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.325497182s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:55:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.325520398s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.394376746s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:54:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.394406461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.246378158s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:53:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.246406822s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.274355621s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:52:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.274397511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.428282869s trace: https://jaeger.dev.storj.io/trace/50b42000841f6169 FAIL (2025-12-09 12:52:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:38:11.658849159 (93h9m51.639201801s) [upload=81864]: Starting upload Dec 9 12:38:11.658950669 (93h9m51.639303312s) [upload=81864]: Done starting upload Dec 9 12:38:11.659075323 (93h9m51.639427958s) [upload=81864]: Uploading segments... Dec 9 12:38:11.659353821 (93h9m51.639706461s) [upload=81864]: Got next segment. Inline: false Dec 9 12:38:11.659419511 (93h9m51.639772133s) [upload=81864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:38:11.659425564 (93h9m51.639778204s) [upload=81864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:38:11.774806210 (93h9m51.755158856s) [seg_pos={0 0}, upload=81864]: Begin upload segment... Dec 9 12:38:11.774914235 (93h9m51.755266879s) [seg_pos={0 0}, upload=81864]: Done begin upload segment. Dec 9 12:38:11.774937614 (93h9m51.755290252s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.774976894 (93h9m51.75532954s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.775596256 (93h9m51.755948902s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776133619 (93h9m51.756486265s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.776703495 (93h9m51.757056143s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777199018 (93h9m51.757551664s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.777663800 (93h9m51.758016446s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778266904 (93h9m51.758619552s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778316661 (93h9m51.758669304s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778684560 (93h9m51.759037202s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.778966742 (93h9m51.759319378s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779259165 (93h9m51.759611804s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779563971 (93h9m51.75991661s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.779854509 (93h9m51.760207148s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.780133583 (93h9m51.760486206s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:11.781738997 (93h9m51.762091627s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Uploading piece... Dec 9 12:38:12.155361633 (93h9m52.135714282s) [noise=true , address=77.90.31.15:4698 , piece_id=KO3I5U5L2I3ITRCSVCNKJAWP4LSDEWGNCUCPCALQQ57VGEZGJOOA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.170417231 (93h9m52.150769879s) [noise=true , address=77.90.31.12:4590 , piece_id=QCSFMBX5KGLNDNEZTRACF63HQEUK5P4F2A7ORMY2XI3XG4TKN2NA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.175183745 (93h9m52.155536393s) [noise=true , address=145.79.227.30:4763 , piece_id=RATTGJ3Y6ME2ZGBDZJY6AIRJIU2S2QVXXD4OT655MZB4Z4D5PRIQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 12:38:12.381789529 (93h9m52.362142181s) [upload=81864]: Next returned nil segment Dec 9 12:38:12.381809355 (93h9m52.362162005s) [upload=81864]: Waiting for error group managing segments... Dec 9 12:38:12.516389161 (93h9m52.496741797s) [noise=true , address=145.79.227.38:4979 , piece_id=GZWFZTXJPIMIIPW5V2FGWZRJBDPW65CGS4V3A2MTQDE3HMSKTAWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.516637496 (93h9m52.496990184s) [noise=true , address=145.79.227.41:4087 , piece_id=C2FJCEAAIZPCDQ3GPWKFDQW43RZ4DCIRZFSGNZ2NC6A2CEBWVEWQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.517626562 (93h9m52.497979199s) [noise=true , address=145.79.227.39:4015 , piece_id=KHXMV7A2PVIZSWI5FIJICDG4BRNUZMBAGQEM3EGHED75V2NUF5IQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522057801 (93h9m52.502410455s) [noise=true , address=145.79.227.27:4689 , piece_id=M3EB3EOXMFWOFEGKHRJTD6GOLMZFUKSSWWZDWKNETIMRP3GOUIMA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.522285261 (93h9m52.502637899s) [noise=true , address=145.79.227.22:4509 , piece_id=2RBUGBZMYUPJC4JHEPQWI6W5BS6AGUUOT2YRXEA5XJJVFVXMHVRA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524577821 (93h9m52.504930465s) [noise=true , address=145.79.227.9:4367 , piece_id=ZZRUGQE2TJAMCSGW7VOXWFHMCJ6JVIBSATYXMUY3IG26COLRLGFQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.524653465 (93h9m52.505006088s) [noise=true , address=145.79.227.40:4049 , piece_id=CBOWX37JZC525GUGFS6NGQ6LQI7O527PHIMTOSPOQHVRLD7L5Q4A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.525273497 (93h9m52.505626141s) [noise=true , address=145.79.227.21:4474 , piece_id=MYTKH5FTSVQ5DEI6XJUFZXX23LABHKUMOC57ZF7VUSC6MVQQIBUA, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.541093670 (93h9m52.52144631s) [noise=true , address=77.90.31.14:4663 , piece_id=56YT27RJBFZRFLSPNHPEJQDT4QOYKWB4SURBUOZELU2QBEEB34KQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.552829059 (93h9m52.533181702s) [noise=true , address=77.90.31.21:4878 , piece_id=5UTWW6F53B36EWNYZ2GMWPRCX5BYXTC6ZENCVGOKQJMR4DCLQHXQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:12.608027592 (93h9m52.58838022s) [noise=true , address=77.90.31.10:4517 , piece_id=TFMBHUPQKDRALBEU3XK6V4BIYYJHAHHKW2LXQJZIA354FSEQZJ7A, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:17.591773437 (93h9m57.572126082s) [noise=true , address=145.79.227.30:4761 , piece_id=7CLFYOPLQPDNHTOYORBXO5PJXHY3BIEUOUQFHFNUOACKBW2BZOOQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: <nil> Dec 9 12:38:41.366723807 (93h10m21.347076455s) [noise=true , address=77.90.31.16:4735 , piece_id=EB74JJDWLS5HJUZLY3OIBAHK25YVXXLYK2UQMAC6FDYHOAHZ6DKQ, seg_pos={0 0}, upload=81864]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:38:41.366795409 (93h10m21.347148037s) [seg_pos={0 0}, upload=81864]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:38:41.473967586 (93h10m21.454320233s) [upload=81864]: Done uploading segments... Dec 9 12:38:41.474033942 (93h10m21.454386592s) [upload=81864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.428311332s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)