Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 354467 reports (Page 572 of 7090)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXXX] duration: 5.000659192s trace: https://jaeger.dev.storj.io/trace/42d1605e2fd17875 FAIL (2026-03-18 20:36:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000669024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX.........................] duration: 11.619873503s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:36:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.619902572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X..X..X...X.............................................] duration: 12.540321692s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:36:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 12.540361379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXXX] duration: 5.00186954s trace: https://jaeger.dev.storj.io/trace/4150d6e4459c68cb FAIL (2026-03-18 20:36:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001887821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX........................] duration: 11.177902456s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:36:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.177951013s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 9.986806876s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:36:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.98683759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX.......................] duration: 11.19823871s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:35:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.198265386s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X..X..X...X............................................] duration: 12.785142895s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:35:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 12.785183667s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 10.162383846s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:35:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.162463755s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXXX] duration: 5.001200756s trace: https://jaeger.dev.storj.io/trace/f6ad17f54b168ef FAIL (2026-03-18 20:35:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001226553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX......................] duration: 11.211012498s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:35:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.211041903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X..X..X...X...........................................] duration: 11.449915676s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:34:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.449949951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX.....................] duration: 11.139858177s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:34:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.140015597s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 11.391149104s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:34:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.39120454s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXXX] duration: 5.001620431s trace: https://jaeger.dev.storj.io/trace/3ef2bfaaff97e923 FAIL (2026-03-18 20:34:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001634142s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X..X..X...X..........................................] duration: 11.764056608s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:33:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.76408824s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 10.675767346s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:33:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.675793524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX....................] duration: 9.132053945s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:33:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.132096997s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXXX] duration: 5.001737428s trace: https://jaeger.dev.storj.io/trace/104ccf9235da45a3 FAIL (2026-03-18 20:33:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001763937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X..X..X...X.........................................] duration: 11.344224549s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:33:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.344346002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X........X..X..X...X........................................] duration: 10.532214953s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:32:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.532253217s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 10.80634105s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:32:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.806389856s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXXX] duration: 5.001998505s trace: https://jaeger.dev.storj.io/trace/1f389acc24b93ad FAIL (2026-03-18 20:32:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002026113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX...................] duration: 8.869166977s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:32:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.869199539s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXXX] duration: 5.001692663s trace: https://jaeger.dev.storj.io/trace/178846c47f706d80 FAIL (2026-03-18 20:32:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001789119s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 10.2432008s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:31:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.243233145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X........X..X..X...X.......................................] duration: 10.565916148s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:31:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.565962561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX..................] duration: 8.238438118s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:31:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.238467958s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXXX] duration: 5.001659111s trace: https://jaeger.dev.storj.io/trace/334813668e03a9e5 FAIL (2026-03-18 20:31:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001762187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X........X..X..X...X......................................] duration: 10.586666665s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:30:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.586695481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX.................] duration: 7.20497338s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:30:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.205005898s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 7.240149773s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:30:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.240230872s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 7.749385227s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:30:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.749508861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..X........X..X..X...X.....................................] duration: 7.691257975s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:30:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.691293731s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXXX] duration: 5.002155184s trace: https://jaeger.dev.storj.io/trace/2c317f14cf7e2c70 FAIL (2026-03-18 20:29:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00218807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 8.323330479s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:29:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.323360906s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX................] duration: 7.206420463s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:29:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.206469893s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XXX] duration: 5.002131622s trace: https://jaeger.dev.storj.io/trace/75770e874382d8fb FAIL (2026-03-18 20:29:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002156964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..X........X..X..X...X....................................] duration: 8.830120538s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:29:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.83013783s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............XX] duration: 5.001180671s trace: https://jaeger.dev.storj.io/trace/8ac63882564de60 FAIL (2026-03-18 20:29:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001214488s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX...............] duration: 6.034656689s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:29:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.03469516s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 7.306413925s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:28:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.306469785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX..............] duration: 5.409320321s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:28:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.409351241s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.730178887s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:28:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.730238382s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............X] duration: 5.001769311s trace: https://jaeger.dev.storj.io/trace/18d9e8ee8d082630 FAIL (2026-03-18 20:28:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001831492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X..X........X..X..X...X...................................] duration: 6.365041656s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:28:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.365070171s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX.............] duration: 2.814439936s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:27:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.814516434s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.875851872s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 20:27:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.875881895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..............] duration: 1.671111329s trace: https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a FAIL (2026-03-18 20:27:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.671129072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..X..X........X..X..X...X..................................] duration: 3.049435097s trace: https://jaeger.dev.storj.io/trace/7f59026f88f99097 FAIL (2026-03-18 20:27:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=188105]: Uploading segments... Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment... Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment. Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece... Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments... Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil> Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments... Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.049465855s core uplink us-select-1 us1-placement-20

1
...
570
571
572
573
574
...
7090