Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXX........................................] duration: 7.792353188s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:49:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.792405109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXX.......................................] duration: 6.977873275s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:48:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.977903716s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001872571s trace: https://jaeger.dev.storj.io/trace/2329bcf9dc505b51 FAIL (2026-03-18 20:48:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001897213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXX......................................] duration: 8.63444642s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:48:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.634472106s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.00081609s trace: https://jaeger.dev.storj.io/trace/2708e2549cda5bb FAIL (2026-03-18 20:48:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000841865s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.002169636s trace: https://jaeger.dev.storj.io/trace/35cb5e25d652c0b7 FAIL (2026-03-18 20:47:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002199422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXX.....................................] duration: 9.77097679s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:47:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.771006934s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001448886s trace: https://jaeger.dev.storj.io/trace/7243725049fa511d FAIL (2026-03-18 20:46:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001473168s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001192428s trace: https://jaeger.dev.storj.io/trace/1ad4576b0ea42ff9 FAIL (2026-03-18 20:46:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001216314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 11.328791883s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:46:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.328821314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXX....................................] duration: 10.514888145s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:46:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.514920749s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001864921s trace: https://jaeger.dev.storj.io/trace/293372aafa991e9c FAIL (2026-03-18 20:46:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001893965s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001959302s trace: https://jaeger.dev.storj.io/trace/7bc6e8062f6bdf84 FAIL (2026-03-18 20:45:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001996473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 11.422767966s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:45:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.422791563s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXX...................................] duration: 11.075659522s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:45:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.07569178s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXX..................................] duration: 8.835289667s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:45:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.835318827s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 11.102942622s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:45:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.102980343s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXX] duration: 5.001906148s trace: https://jaeger.dev.storj.io/trace/1650ec64f5283dd3 FAIL (2026-03-18 20:45:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001934725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 10.68650615s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:44:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.686542724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXX.................................] duration: 10.923861682s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:43:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.923889928s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXX] duration: 5.002309914s trace: https://jaeger.dev.storj.io/trace/46ea2984c5a4e4e8 FAIL (2026-03-18 20:43:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002342751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...X.......................................................] duration: 10.116025096s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:43:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.116128627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXX] duration: 5.000762576s trace: https://jaeger.dev.storj.io/trace/693c9c4417026cd2 FAIL (2026-03-18 20:43:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00078819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...X......................................................] duration: 11.57009624s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:43:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.570131438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XXX................................] duration: 8.99662033s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:42:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.996643692s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXX] duration: 5.000750455s trace: https://jaeger.dev.storj.io/trace/148035eb58feb31b FAIL (2026-03-18 20:42:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000780063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration: 9.751269623s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:42:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.751299899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXX...............................] duration: 10.506853929s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:42:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.506950845s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..X...X....................................................] duration: 11.291838141s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:42:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.291945408s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXX] duration: 5.000731238s trace: https://jaeger.dev.storj.io/trace/48080a7a0975e6b6 FAIL (2026-03-18 20:41:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000754338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX..............................] duration: 10.565416742s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:41:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.565449344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX.............................] duration: 10.977549587s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:40:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.977581218s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..X...X...................................................] duration: 12.137800815s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:40:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 12.137838152s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXX] duration: 5.00221987s trace: https://jaeger.dev.storj.io/trace/47a41117cdd38415 FAIL (2026-03-18 20:40:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002257288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..X...X..................................................] duration: 11.564121639s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:40:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.564146623s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXXX] duration: 5.001623607s trace: https://jaeger.dev.storj.io/trace/773827b32f24ec06 FAIL (2026-03-18 20:39:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00165154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX............................] duration: 10.694507627s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:39:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.694536262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..X..X...X.................................................] duration: 9.82320769s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:39:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.823244895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXXX] duration: 5.000687855s trace: https://jaeger.dev.storj.io/trace/6b28aee9b24e63c2 FAIL (2026-03-18 20:39:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000712272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 10.489369839s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:39:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.489400555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX...........................] duration: 10.791098664s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:39:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.791127879s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 10.837528227s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:38:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.837562881s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXXX] duration: 5.000658087s trace: https://jaeger.dev.storj.io/trace/23703af246b9a1cf FAIL (2026-03-18 20:38:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000684641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..X..X...X................................................] duration: 11.556427525s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:38:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.556517754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X..X..X...X...............................................] duration: 11.388261232s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:37:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.3882899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX..........................] duration: 11.04480893s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:37:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.044841213s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXXX] duration: 5.001877709s trace: https://jaeger.dev.storj.io/trace/48ed2802dab9db5a FAIL (2026-03-18 20:37:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001907304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 11.273879109s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:37:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.273914686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X..X..X...X..............................................] duration: 11.543898323s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:37:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.543930543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 11.898377539s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:36:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.89843972s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)