Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX.] duration:
2.411921046s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:18:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.411952653s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX...] duration:
2.031191492s trace:
https://jaeger.dev.storj.io/trace/778e2ecfec472f01
FAIL (2026-03-18 20:18:25 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.031227474s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [......X..X...X..X..X........X..X..X...X.....................] duration:
3.128877271s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:18:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.128912088s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [......XXX.XXX...............X...............................] duration:
3.735786261s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:18:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.73581825s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX..] duration:
1.688416147s trace:
https://jaeger.dev.storj.io/trace/778e2ecfec472f01
FAIL (2026-03-18 20:17:54 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.688513248s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration:
30.108766027s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:17:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.108855791s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.......X..X...X..X..X........X..X..X...X....................] duration:
3.180047292s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:17:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.180073165s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.......XXX.XXX...............X..............................] duration:
2.860080316s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:17:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.860125321s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [.XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.] duration:
1.76722402s trace:
https://jaeger.dev.storj.io/trace/778e2ecfec472f01
FAIL (2026-03-18 20:17:15 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.767253406s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [..XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration:
5.001991078s trace:
https://jaeger.dev.storj.io/trace/778e2ecfec472f01
FAIL (2026-03-18 20:17:00 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002015063s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [........X..X...X..X..X........X..X..X...X...................] duration:
9.745346428s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:16:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 9.745377421s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [........XXX.XXX...............X.............................] duration:
12.387050756s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:16:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.387145579s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
52 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration:
30.107799538s trace:
https://jaeger.dev.storj.io/trace/5b9f114c9946a871
FAIL (2026-03-18 20:16:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:16:06.380306210 (196h5m9.395975819s) [upload=188349]: Starting upload
Mar 18 20:16:06.380397769 (196h5m9.396067364s) [upload=188349]: Done starting upload
Mar 18 20:16:06.380779943 (196h5m9.396449544s) [upload=188349]: Uploading segments...
Mar 18 20:16:06.380842471 (196h5m9.396512078s) [upload=188349]: Got next segment. Inline: false
Mar 18 20:16:06.380895159 (196h5m9.396564769s) [upload=188349]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:16:06.380902841 (196h5m9.396572448s) [upload=188349]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:16:09.150906946 (196h5m12.166576591s) [seg_pos={0 0}, upload=188349]: Begin upload segment...
Mar 18 20:16:09.150993328 (196h5m12.166662967s) [seg_pos={0 0}, upload=188349]: Done begin upload segment.
Mar 18 20:16:09.151050153 (196h5m12.166719783s) [noise=true , address=77.90.31.12:4594 , piece_id=FPVGRZNCOVMQW3A7KK2KJIV5SCFEEYT5TYLUD7WLKPMGYDENELJQ, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.152610132 (196h5m12.168279757s) [noise=true , address=77.90.31.16:4735 , piece_id=AYYEZEQ64QNJYKWXLN6TVHOKGKHQV7UJUD3CTNIHHHXZUKNGDDSA, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.153203580 (196h5m12.168873195s) [noise=true , address=77.90.31.19:4845 , piece_id=KDFIXTGX625KPE5YM6LAEER2RKATJYPZRXZOEEKJMNDF4SOP5SHQ, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.153890604 (196h5m12.169560222s) [noise=true , address=45.91.116.13:4156 , piece_id=AVVAFEAVWZUCM7Q6KLHO5IKG4OD4JT5CLHPFLASZF6WTATFWYILA, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.154444386 (196h5m12.17011401s) [noise=true , address=79.127.219.46:4894 , piece_id=MADQSNU5XXGRPTHYBX5YMDVNNLHHLPBXK2RAQTVP6C4PXKUZDAJA, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.154897514 (196h5m12.170567126s) [noise=true , address=45.91.116.7:4009 , piece_id=FSWN7P4KUJW46MEI436EGNICJQMDR5FDC5VDSJWNAVRO32UVLE6A, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.155367194 (196h5m12.171036811s) [noise=true , address=79.127.219.38:4819 , piece_id=NKAGLULOORVBSTIOPHHXZG35HBS5GRY3KFN45EWNHJHZ3BHQZDRQ, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.155861340 (196h5m12.171530952s) [noise=true , address=77.90.31.14:4662 , piece_id=BWVE7CDXSDYDGEEIIZFS2RYRM5QNFN2C4F2VB2EDVYW73UZ4VFGA, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.156559047 (196h5m12.172228649s) [noise=true , address=145.79.227.26:4656 , piece_id=245XOGAI3CMBUE77PXYFVSGNAX54D76SYKYRNKH2JLHWYUZAD5DQ, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.157475778 (196h5m12.173145391s) [noise=true , address=77.90.31.18:4809 , piece_id=VPEA5ZOOC2Q6SEP2IMIPFOYB5IL563QQ5JHBKSPHPYOP3OX6CLBQ, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.158138834 (196h5m12.173808439s) [noise=true , address=145.79.227.39:4017 , piece_id=AA5U76AOFLYLABQ4AI74WPD3DZFXIF34HKBWKFWYMM5TIRIKQTIA, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.158717971 (196h5m12.174387587s) [noise=true , address=45.91.116.5:4300 , piece_id=YEGL6JAVQ5IWBYIIWJKXRHJWJXWACHCLIXG6SESX3EKLJNAUCVHQ, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.159374860 (196h5m12.175044474s) [noise=true , address=45.91.116.12:4226 , piece_id=TSHH4O6ASYT5LO4CULYNPHQKHY6TQD3GQBJXNWP6HZJK5GFW2UQA, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.160021304 (196h5m12.175690927s) [noise=true , address=145.79.227.41:4090 , piece_id=PSHMTVWJ5YTCIQ5RJCK2L2PITMD2NMPSGLZ6FI5ZYGZAUTGS7VFQ, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.160623593 (196h5m12.176293208s) [noise=true , address=77.90.31.13:4626 , piece_id=W5H7OKQZKIGHUJ2VVQE4RB7SVBJF3OGCGOJ2QJ6VGGVTG2FDJTWQ, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.161104007 (196h5m12.176773627s) [noise=true , address=145.79.227.33:4872 , piece_id=A7VJZ7LGYY3G7S4WOWYGMX2JYWQ2WSYMEFVWYG27NDEOGLXA4TNQ, seg_pos={0 0}, upload=188349]: Uploading piece...
Mar 18 20:16:09.779390271 (196h5m12.795059875s) [upload=188349]: Next returned nil segment
Mar 18 20:16:09.779408229 (196h5m12.795077839s) [upload=188349]: Waiting for error group managing segments...
Mar 18 20:16:09.914776266 (196h5m12.930445867s) [noise=true , address=77.90.31.18:4809 , piece_id=VPEA5ZOOC2Q6SEP2IMIPFOYB5IL563QQ5JHBKSPHPYOP3OX6CLBQ, seg_pos={0 0}, upload=188349]: Done uploading piece. err: <nil>
Mar 18 20:16:09.914897729 (196h5m12.93056733s) [noise=true , address=77.90.31.14:4662 , piece_id=BWVE7CDXSDYDGEEIIZFS2RYRM5QNFN2C4F2VB2EDVYW73UZ4VFGA, seg_pos={0 0}, upload=188349]: Done uploading piece. err: <nil>
Mar 18 20:16:09.925323624 (196h5m12.940993225s) [noise=true , address=79.127.219.46:4894 , piece_id=MADQSNU5XXGRPTHYBX5YMDVNNLHHLPBXK2RAQTVP6C4PXKUZDAJA, seg_pos={0 0}, upload=188349]: Done uploading piece. err: <nil>
Mar 18 20:16:09.926323834 (196h5m12.941993434s) [noise=true , address=45.91.116.13:4156 , piece_id=AVVAFEAVWZUCM7Q6KLHO5IKG4OD4JT5CLHPFLASZF6WTATFWYILA, seg_pos={0 0}, upload=188349]: Done uploading piece. err: <nil>
Mar 18 20:16:09.936514921 (196h5m12.952184527s) [noise=true , address=77.90.31.16:4735 , piece_id=AYYEZEQ64QNJYKWXLN6TVHOKGKHQV7UJUD3CTNIHHHXZUKNGDDSA, seg_pos={0 0}, upload=188349]: Done uploading piece. err: <nil>
Mar 18 20:16:09.942889240 (196h5m12.95855884s) [noise=true , address=45.91.116.7:4009 , piece_id=FSWN7P4KUJW46MEI436EGNICJQMDR5FDC5VDSJWNAVRO32UVLE6A, seg_pos={0 0}, upload=188349]: Done uploading piece. err: <nil>
Mar 18 20:16:09.946756495 (196h5m12.962426096s) [noise=true , address=45.91.116.5:4300 , piece_id=YEGL6JAVQ5IWBYIIWJKXRHJWJXWACHCLIXG6SESX3EKLJNAUCVHQ, seg_pos={0 0}, upload=188349]: Done uploading piece. err: <nil>
Mar 18 20:16:09.947642422 (196h5m12.963312022s) [noise=true , address=77.90.31.13:4626 , piece_id=W5H7OKQZKIGHUJ2VVQE4RB7SVBJF3OGCGOJ2QJ6VGGVTG2FDJTWQ, seg_pos={0 0}, upload=188349]: Done uploading piece. err: <nil>
Mar 18 20:16:09.948425587 (196h5m12.964095188s) [noise=true , address=77.90.31.12:4594 , piece_id=FPVGRZNCOVMQW3A7KK2KJIV5SCFEEYT5TYLUD7WLKPMGYDENELJQ, seg_pos={0 0}, upload=188349]: Done uploading piece. err: <nil>
Mar 18 20:16:09.956181228 (196h5m12.971850832s) [noise=true , address=45.91.116.12:4226 , piece_id=TSHH4O6ASYT5LO4CULYNPHQKHY6TQD3GQBJXNWP6HZJK5GFW2UQA, seg_pos={0 0}, upload=188349]: Done uploading piece. err: <nil>
Mar 18 20:16:09.959627276 (196h5m12.975296886s) [noise=true , address=77.90.31.19:4845 , piece_id=KDFIXTGX625KPE5YM6LAEER2RKATJYPZRXZOEEKJMNDF4SOP5SHQ, seg_pos={0 0}, upload=188349]: Done uploading piece. err: <nil>
Mar 18 20:16:10.004720535 (196h5m13.020390168s) [noise=true , address=79.127.219.38:4819 , piece_id=NKAGLULOORVBSTIOPHHXZG35HBS5GRY3KFN45EWNHJHZ3BHQZDRQ, seg_pos={0 0}, upload=188349]: Done uploading piece. err: <nil>
Mar 18 20:16:36.096048077 (196h5m39.111717684s) [noise=true , address=145.79.227.41:4090 , piece_id=PSHMTVWJ5YTCIQ5RJCK2L2PITMD2NMPSGLZ6FI5ZYGZAUTGS7VFQ, seg_pos={0 0}, upload=188349]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:145.79.227.41:4090): protocol: rpc: dial tcp 145.79.227.41:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:16:36.096833856 (196h5m39.112503469s) [noise=true , address=145.79.227.33:4872 , piece_id=A7VJZ7LGYY3G7S4WOWYGMX2JYWQ2WSYMEFVWYG27NDEOGLXA4TNQ, seg_pos={0 0}, upload=188349]: Done uploading piece. err: ecclient: upload failed (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec, address:145.79.227.33:4872): protocol: rpc: dial tcp 145.79.227.33:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:16:36.097310688 (196h5m39.112980299s) [noise=true , address=145.79.227.26:4656 , piece_id=245XOGAI3CMBUE77PXYFVSGNAX54D76SYKYRNKH2JLHWYUZAD5DQ, seg_pos={0 0}, upload=188349]: Done uploading piece. err: ecclient: upload failed (node:1u6e2ZjrPg2q3EDN2ZjDj9hzEmCAoRvt39HMeM79gcVmMVm53j, address:145.79.227.26:4656): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:16:36.097876493 (196h5m39.113546099s) [noise=true , address=145.79.227.39:4017 , piece_id=AA5U76AOFLYLABQ4AI74WPD3DZFXIF34HKBWKFWYMM5TIRIKQTIA, seg_pos={0 0}, upload=188349]: Done uploading piece. err: ecclient: upload failed (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk, address:145.79.227.39:4017): protocol: rpc: dial tcp 145.79.227.39:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:16:36.097997558 (196h5m39.11366716s) [seg_pos={0 0}, upload=188349]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:16:36.201891632 (196h5m39.217561231s) [upload=188349]: Done uploading segments...
Mar 18 20:16:36.201978524 (196h5m39.217648125s) [upload=188349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.10782682s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.........XXX.XXX...............X............................] duration:
17.911668098s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:16:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.911721197s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [...XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration:
5.000743824s trace:
https://jaeger.dev.storj.io/trace/6aca26131a09d5db
FAIL (2026-03-18 20:16:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000773035s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [..........XXX.XXX...............X...........................] duration:
16.42319567s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:16:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 16.423243929s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
51 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.X] duration:
30.111629377s trace:
https://jaeger.dev.storj.io/trace/64b7d0f99d4135f5
FAIL (2026-03-18 20:15:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Mar 18 20:15:16.236174585 (196h4m19.251844189s) [upload=188334]: Starting upload
Mar 18 20:15:16.236250323 (196h4m19.251919934s) [upload=188334]: Done starting upload
Mar 18 20:15:16.236563159 (196h4m19.252232754s) [upload=188334]: Uploading segments...
Mar 18 20:15:16.236630233 (196h4m19.252299836s) [upload=188334]: Got next segment. Inline: false
Mar 18 20:15:16.236677193 (196h4m19.252346793s) [upload=188334]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:15:16.236682166 (196h4m19.25235177s) [upload=188334]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:15:19.043977479 (196h4m22.05964708s) [seg_pos={0 0}, upload=188334]: Begin upload segment...
Mar 18 20:15:19.044059348 (196h4m22.059728955s) [seg_pos={0 0}, upload=188334]: Done begin upload segment.
Mar 18 20:15:19.044115242 (196h4m22.059784856s) [noise=true , address=77.90.31.21:4880 , piece_id=MMYFJZTI7AC6VH4BFXCMPUGH5N7N3H7AAEN5J7HDCMTZYRHA4KBQ, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.044682809 (196h4m22.060352419s) [noise=true , address=145.79.227.24:4585 , piece_id=7JUIRTD6PRLJP2BSOMIIYYF5X6VQPTL7OUAQ6JU6ABHGDORWZXSA, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.045199806 (196h4m22.060869411s) [noise=true , address=45.91.116.14:4216 , piece_id=CSOSXJFLZDCHDZ4VDNEVEIBI7D6SPQD3WVZZ7EIIPG5V22HLTJEQ, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.045720379 (196h4m22.061389982s) [noise=true , address=145.79.227.44:4194 , piece_id=OUQ25ED5WQLXJMD3O2GLPQAUDH5WX5PJJNPAUYIZNNR5ZV42TMDA, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.046218843 (196h4m22.061888446s) [noise=true , address=45.91.116.5:4298 , piece_id=WIIPET62WPIJ4XCHUS4O7BQ2MRUYBOOXZWX2UOE2OEEV7HGRNTYQ, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.046668810 (196h4m22.062338415s) [noise=true , address=77.90.31.21:4878 , piece_id=H4BD3UGFUV73TSYELPL3AJS3LE32TWXYPD4N6YS3Y473WGMW7QNQ, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.046985233 (196h4m22.062654843s) [noise=true , address=77.90.31.11:4554 , piece_id=EHNEGGMFMI26KLCPUZNDZ4CK5YHGOWY4JAHIS2TF53ITFQLHS4XA, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.047101656 (196h4m22.062771262s) [noise=true , address=45.91.116.9:4261 , piece_id=V7E673VAW7XYYZCJHBKVKGFJZOR3AFZYGRBQV66FCK5HVBG6QKOQ, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.047457259 (196h4m22.063126868s) [noise=true , address=145.79.227.42:4126 , piece_id=HLY73BUWICUWY5LQT4XGLR3FV7FPB77PGRIOIH2UAQJFPXUN5HTQ, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.047519347 (196h4m22.063189014s) [noise=true , address=45.91.116.10:4081 , piece_id=5XHFN7WWPE7KTVV7XNGSQSYLOHIFKHJ4QXBRUBNP3FHDBEVO7KGQ, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.048003480 (196h4m22.063673081s) [noise=true , address=77.90.31.19:4846 , piece_id=SHBHZ7VRGSFIVS7FEPYI2KNJXHALQRSX22MJMCM7VJFZE5DNF2AA, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.048038139 (196h4m22.063707749s) [noise=true , address=45.91.116.8:4080 , piece_id=TA5R5JALOGCJBWCZ2527Z5NBSZ2CDPOLTUVDZORFF3TVZXK6JYZQ, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.048399533 (196h4m22.064069133s) [noise=true , address=79.127.219.46:4893 , piece_id=YZLJGHHQ7TV63TA4TS4CDKABRFEWLE2DELFX67TUYAAGI7CCBZNQ, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.048622041 (196h4m22.064291654s) [noise=true , address=79.127.219.41:4929 , piece_id=OMWF3EPMYAOP5N5L5DDVNW5627NA2ILKF7Q4HPB7WIVP7CXMJBWQ, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.049326470 (196h4m22.064996082s) [noise=true , address=145.79.227.32:4838 , piece_id=5GX3TA6GGFYJP4MTCAYD7P2DI7KI57E4SF56IABMAN4PLOTXEVCQ, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.052365661 (196h4m22.068035367s) [noise=true , address=145.79.227.43:4160 , piece_id=RKHANKI5UIPGVE4BHIFGVFB3GTXBCHEWNPMO6PCI442GC4FEUDKA, seg_pos={0 0}, upload=188334]: Uploading piece...
Mar 18 20:15:19.653001666 (196h4m22.668671284s) [upload=188334]: Next returned nil segment
Mar 18 20:15:19.653021178 (196h4m22.668690787s) [upload=188334]: Waiting for error group managing segments...
Mar 18 20:15:19.776942036 (196h4m22.792611643s) [noise=true , address=77.90.31.21:4878 , piece_id=H4BD3UGFUV73TSYELPL3AJS3LE32TWXYPD4N6YS3Y473WGMW7QNQ, seg_pos={0 0}, upload=188334]: Done uploading piece. err: <nil>
Mar 18 20:15:19.776994699 (196h4m22.792664299s) [noise=true , address=77.90.31.11:4554 , piece_id=EHNEGGMFMI26KLCPUZNDZ4CK5YHGOWY4JAHIS2TF53ITFQLHS4XA, seg_pos={0 0}, upload=188334]: Done uploading piece. err: <nil>
Mar 18 20:15:19.779225445 (196h4m22.794895049s) [noise=true , address=77.90.31.21:4880 , piece_id=MMYFJZTI7AC6VH4BFXCMPUGH5N7N3H7AAEN5J7HDCMTZYRHA4KBQ, seg_pos={0 0}, upload=188334]: Done uploading piece. err: <nil>
Mar 18 20:15:19.787661564 (196h4m22.803331167s) [noise=true , address=45.91.116.14:4216 , piece_id=CSOSXJFLZDCHDZ4VDNEVEIBI7D6SPQD3WVZZ7EIIPG5V22HLTJEQ, seg_pos={0 0}, upload=188334]: Done uploading piece. err: <nil>
Mar 18 20:15:19.788311078 (196h4m22.803980678s) [noise=true , address=45.91.116.10:4081 , piece_id=5XHFN7WWPE7KTVV7XNGSQSYLOHIFKHJ4QXBRUBNP3FHDBEVO7KGQ, seg_pos={0 0}, upload=188334]: Done uploading piece. err: <nil>
Mar 18 20:15:19.792607466 (196h4m22.808277068s) [noise=true , address=45.91.116.8:4080 , piece_id=TA5R5JALOGCJBWCZ2527Z5NBSZ2CDPOLTUVDZORFF3TVZXK6JYZQ, seg_pos={0 0}, upload=188334]: Done uploading piece. err: <nil>
Mar 18 20:15:19.793783798 (196h4m22.809453413s) [noise=true , address=77.90.31.19:4846 , piece_id=SHBHZ7VRGSFIVS7FEPYI2KNJXHALQRSX22MJMCM7VJFZE5DNF2AA, seg_pos={0 0}, upload=188334]: Done uploading piece. err: <nil>
Mar 18 20:15:19.797046200 (196h4m22.812715803s) [noise=true , address=45.91.116.9:4261 , piece_id=V7E673VAW7XYYZCJHBKVKGFJZOR3AFZYGRBQV66FCK5HVBG6QKOQ, seg_pos={0 0}, upload=188334]: Done uploading piece. err: <nil>
Mar 18 20:15:19.798454921 (196h4m22.814124531s) [noise=true , address=79.127.219.41:4929 , piece_id=OMWF3EPMYAOP5N5L5DDVNW5627NA2ILKF7Q4HPB7WIVP7CXMJBWQ, seg_pos={0 0}, upload=188334]: Done uploading piece. err: <nil>
Mar 18 20:15:19.820799207 (196h4m22.83646881s) [noise=true , address=45.91.116.5:4298 , piece_id=WIIPET62WPIJ4XCHUS4O7BQ2MRUYBOOXZWX2UOE2OEEV7HGRNTYQ, seg_pos={0 0}, upload=188334]: Done uploading piece. err: <nil>
Mar 18 20:15:19.854645724 (196h4m22.870315331s) [noise=true , address=79.127.219.46:4893 , piece_id=YZLJGHHQ7TV63TA4TS4CDKABRFEWLE2DELFX67TUYAAGI7CCBZNQ, seg_pos={0 0}, upload=188334]: Done uploading piece. err: <nil>
Mar 18 20:15:45.943413533 (196h4m48.959083144s) [noise=true , address=145.79.227.32:4838 , piece_id=5GX3TA6GGFYJP4MTCAYD7P2DI7KI57E4SF56IABMAN4PLOTXEVCQ, seg_pos={0 0}, upload=188334]: Done uploading piece. err: ecclient: upload failed (node:1eorQLNmgEaXv8myr6t3Q8rysGwsvtdZzu5b7sndhoANb6Waj6, address:145.79.227.32:4838): protocol: rpc: dial tcp 145.79.227.32:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:15:45.943726239 (196h4m48.959395851s) [noise=true , address=145.79.227.42:4126 , piece_id=HLY73BUWICUWY5LQT4XGLR3FV7FPB77PGRIOIH2UAQJFPXUN5HTQ, seg_pos={0 0}, upload=188334]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: rpc: dial tcp 145.79.227.42:4126: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:15:45.943774135 (196h4m48.959443743s) [noise=true , address=145.79.227.43:4160 , piece_id=RKHANKI5UIPGVE4BHIFGVFB3GTXBCHEWNPMO6PCI442GC4FEUDKA, seg_pos={0 0}, upload=188334]: Done uploading piece. err: ecclient: upload failed (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84, address:145.79.227.43:4160): protocol: rpc: dial tcp 145.79.227.43:4160: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:15:45.943821435 (196h4m48.959491046s) [noise=true , address=145.79.227.24:4585 , piece_id=7JUIRTD6PRLJP2BSOMIIYYF5X6VQPTL7OUAQ6JU6ABHGDORWZXSA, seg_pos={0 0}, upload=188334]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: rpc: dial tcp 145.79.227.24:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:15:45.944048030 (196h4m48.959717637s) [noise=true , address=145.79.227.44:4194 , piece_id=OUQ25ED5WQLXJMD3O2GLPQAUDH5WX5PJJNPAUYIZNNR5ZV42TMDA, seg_pos={0 0}, upload=188334]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: rpc: dial tcp 145.79.227.44:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:15:45.944086641 (196h4m48.959756231s) [seg_pos={0 0}, upload=188334]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Mar 18 20:15:46.052152319 (196h4m49.067821926s) [upload=188334]: Done uploading segments...
Mar 18 20:15:46.052212030 (196h4m49.06788164s) [upload=188334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.111674414s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [....XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration:
5.001510432s trace:
https://jaeger.dev.storj.io/trace/6f7be684ecb63567
FAIL (2026-03-18 20:15:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001534249s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.........X..X...X..X..X........X..X..X...X..................] duration:
3.570382349s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:15:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.570416298s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [...........XXX.XXX...............X..........................] duration:
15.108529235s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:15:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.108558009s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [..........X..X...X..X..X........X..X..X...X.................] duration:
15.713103496s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:15:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 15.71313014s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
50 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.] duration:
18.921412397s trace:
https://jaeger.dev.storj.io/trace/4df8f6538811f766
FAIL (2026-03-18 20:14:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Mar 18 20:13:14.354937904 (196h2m17.37060751s) [upload=188305]: Starting upload
Mar 18 20:13:14.355011049 (196h2m17.370680639s) [upload=188305]: Done starting upload
Mar 18 20:13:14.355299147 (196h2m17.370968736s) [upload=188305]: Uploading segments...
Mar 18 20:13:14.355370443 (196h2m17.371040054s) [upload=188305]: Got next segment. Inline: false
Mar 18 20:13:14.355409298 (196h2m17.371078901s) [upload=188305]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:13:14.355430832 (196h2m17.371100435s) [upload=188305]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:13:16.834958230 (196h2m19.850627831s) [seg_pos={0 0}, upload=188305]: Begin upload segment...
Mar 18 20:13:16.835023997 (196h2m19.850693597s) [seg_pos={0 0}, upload=188305]: Done begin upload segment.
Mar 18 20:13:16.835064763 (196h2m19.850734364s) [noise=true , address=45.91.116.7:4012 , piece_id=UQW7DYUETWXG5GKVYBKIP376YLS2YU2GULVS4UIJFIP7SIQGEB6Q, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.835621867 (196h2m19.85129147s) [noise=true , address=145.79.227.31:4803 , piece_id=TI7GIJYALNLLGSW7UMALOZQLPEPEALEBJYVJGJXYVAYNXGRDEHOA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.835848947 (196h2m19.851518554s) [noise=true , address=45.91.116.13:4188 , piece_id=Y4RTYNLI6TTDBJP5TDTNUP6AJBM7MGHK7A2XFI7SIUHX5B57BCTQ, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.836235001 (196h2m19.85190461s) [noise=true , address=77.90.31.10:4522 , piece_id=XE56P6BNXPFGPENHPNTB2BHXV42SX4HYSWDY3FOY5FNPKZQAYTJA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.836984877 (196h2m19.852654486s) [noise=true , address=77.90.31.12:4594 , piece_id=S6U7OMFLHLY2QC6IDUTATM2N7OVN2UXDBSHVDUZHU62YOSAUR26Q, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.838283498 (196h2m19.853953104s) [noise=true , address=145.79.227.44:4197 , piece_id=IVHM5W7JXKU7KDSRIPBMBHMQK3ELPQLPRMO2MZ2FKORG5RGYTWGA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.838349018 (196h2m19.854018621s) [noise=true , address=145.79.227.24:4585 , piece_id=TEPYUD4BHMCHFRP5ROENFFW6EC5SSKZS4OJ26UUZ72OG3UWUVNLA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.839075222 (196h2m19.85474483s) [noise=true , address=145.79.227.34:4910 , piece_id=QLSA5WE5XHOQ6VR6QLEOZUFX3ZLJPMGVG4JZXZ2OMV2C6HLUCFTA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.839266483 (196h2m19.854936085s) [noise=true , address=145.79.227.41:4087 , piece_id=NTPD2S4Q3XGXC4UN7KALS2S4POJ22FWDP3UZPTBT2Y65FESVX67Q, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.839925017 (196h2m19.855594616s) [noise=true , address=45.91.116.5:4332 , piece_id=IV2ANOHFT6ISMBMUIJXK32GHE27CD64R3NWTBFDPUEWX3I6S3HMQ, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.839972036 (196h2m19.855641642s) [noise=true , address=145.79.227.20:4441 , piece_id=WGHNTAQ35B4NMZJENGMKG6TJMJIRBBFDQPNFG5RJDTWAMA7FFQYQ, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.840590665 (196h2m19.856260271s) [noise=true , address=45.91.116.11:4118 , piece_id=S3AMOOH6ANSNN4JTWZV5N2AHJ756SBFOYIGJ6EIYQOXNYYVQ2MCA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.840800863 (196h2m19.856470466s) [noise=true , address=45.91.116.8:4046 , piece_id=KO7KB47XGAEUEGAYLAAL7AJMZSJMMDMC67AAIHUEKKLMGSCYOGGA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.843419599 (196h2m19.859089194s) [noise=true , address=145.79.227.32:4837 , piece_id=MLXDI3OPZLO465PF3V4OJYKWQZLJFLR7IBE6UCG2UNTJWWBQU22Q, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.844229690 (196h2m19.859899283s) [noise=true , address=45.91.116.10:4085 , piece_id=JM35UZHZYJYCRO66IULRQNKUVGM6MFDVT73B7X2BQ6XKWL4ETMVA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.844937348 (196h2m19.860606935s) [noise=true , address=45.91.116.6:4989 , piece_id=U6XHKJNRWJQ7FXP3NYHS4LIQJFMXKR7MTQOH2Y5R6L3UUUJQC3ZA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:17.434373901 (196h2m20.450043491s) [upload=188305]: Next returned nil segment
Mar 18 20:13:17.434389632 (196h2m20.450059241s) [upload=188305]: Waiting for error group managing segments...
Mar 18 20:13:17.575633989 (196h2m20.591303601s) [noise=true , address=45.91.116.7:4012 , piece_id=UQW7DYUETWXG5GKVYBKIP376YLS2YU2GULVS4UIJFIP7SIQGEB6Q, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.581533247 (196h2m20.597202858s) [noise=true , address=45.91.116.11:4118 , piece_id=S3AMOOH6ANSNN4JTWZV5N2AHJ756SBFOYIGJ6EIYQOXNYYVQ2MCA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.582667981 (196h2m20.598337591s) [noise=true , address=77.90.31.10:4522 , piece_id=XE56P6BNXPFGPENHPNTB2BHXV42SX4HYSWDY3FOY5FNPKZQAYTJA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.584090464 (196h2m20.599760072s) [noise=true , address=45.91.116.5:4332 , piece_id=IV2ANOHFT6ISMBMUIJXK32GHE27CD64R3NWTBFDPUEWX3I6S3HMQ, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.584745231 (196h2m20.600414839s) [noise=true , address=45.91.116.10:4085 , piece_id=JM35UZHZYJYCRO66IULRQNKUVGM6MFDVT73B7X2BQ6XKWL4ETMVA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.588416935 (196h2m20.604086541s) [noise=true , address=45.91.116.6:4989 , piece_id=U6XHKJNRWJQ7FXP3NYHS4LIQJFMXKR7MTQOH2Y5R6L3UUUJQC3ZA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.595781267 (196h2m20.61145087s) [noise=true , address=45.91.116.13:4188 , piece_id=Y4RTYNLI6TTDBJP5TDTNUP6AJBM7MGHK7A2XFI7SIUHX5B57BCTQ, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.605473159 (196h2m20.621142768s) [noise=true , address=45.91.116.8:4046 , piece_id=KO7KB47XGAEUEGAYLAAL7AJMZSJMMDMC67AAIHUEKKLMGSCYOGGA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.609297617 (196h2m20.624967223s) [noise=true , address=77.90.31.12:4594 , piece_id=S6U7OMFLHLY2QC6IDUTATM2N7OVN2UXDBSHVDUZHU62YOSAUR26Q, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:44.054807254 (196h2m47.070476862s) [noise=true , address=145.79.227.24:4585 , piece_id=TEPYUD4BHMCHFRP5ROENFFW6EC5SSKZS4OJ26UUZ72OG3UWUVNLA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: rpc: dial tcp 145.79.227.24:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.054871399 (196h2m47.070541008s) [noise=true , address=145.79.227.32:4837 , piece_id=MLXDI3OPZLO465PF3V4OJYKWQZLJFLR7IBE6UCG2UNTJWWBQU22Q, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:1jmioLYNGZjDcByx8B1oHbqfid2DRqRfgoHhz85VcoTqbWz6Ls, address:145.79.227.32:4837): protocol: rpc: dial tcp 145.79.227.32:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.054881694 (196h2m47.070551288s) [noise=true , address=145.79.227.34:4910 , piece_id=QLSA5WE5XHOQ6VR6QLEOZUFX3ZLJPMGVG4JZXZ2OMV2C6HLUCFTA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:12bRYzNggkq7QWFTP9fB94dTrMMX8AYvKMt2zjmJBzT1VwqyPVZ, address:145.79.227.34:4910): protocol: rpc: dial tcp 145.79.227.34:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.054927602 (196h2m47.070597212s) [noise=true , address=145.79.227.41:4087 , piece_id=NTPD2S4Q3XGXC4UN7KALS2S4POJ22FWDP3UZPTBT2Y65FESVX67Q, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: rpc: dial tcp 145.79.227.41:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.054966119 (196h2m47.070635734s) [noise=true , address=145.79.227.44:4197 , piece_id=IVHM5W7JXKU7KDSRIPBMBHMQK3ELPQLPRMO2MZ2FKORG5RGYTWGA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:145.79.227.44:4197): protocol: rpc: dial tcp 145.79.227.44:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.055032355 (196h2m47.070701967s) [noise=true , address=145.79.227.20:4441 , piece_id=WGHNTAQ35B4NMZJENGMKG6TJMJIRBBFDQPNFG5RJDTWAMA7FFQYQ, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: rpc: dial tcp 145.79.227.20:4441: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.055083370 (196h2m47.070752971s) [noise=true , address=145.79.227.31:4803 , piece_id=TI7GIJYALNLLGSW7UMALOZQLPEPEALEBJYVJGJXYVAYNXGRDEHOA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:12GFk3GkhB88UKQdZM8vZ9VLNuuYDrS5ThZjVAdbL2CtTQVMDEB, address:145.79.227.31:4803): protocol: rpc: dial tcp 145.79.227.31:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.055107374 (196h2m47.070776974s) [seg_pos={0 0}, upload=188305]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Mar 18 20:13:44.168234368 (196h2m47.183903969s) [upload=188305]: Done uploading segments...
Mar 18 20:13:44.168271973 (196h2m47.183941579s) [upload=188305]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 18.921440492s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [.....XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration:
5.000831235s trace:
https://jaeger.dev.storj.io/trace/32591e83b9978a50
FAIL (2026-03-18 20:14:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000872193s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [............XXX.XXX...............X.........................] duration:
19.60462764s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:14:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.604656379s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [...........X..X...X..X..X........X..X..X...X................] duration:
3.486388049s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:14:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.486443401s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [......XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001208528s trace:
https://jaeger.dev.storj.io/trace/32591e83b9978a50
FAIL (2026-03-18 20:14:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001232881s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [............X..X...X..X..X........X..X..X...X...............] duration:
17.656412997s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:13:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 17.656449606s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
50 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.115541291s trace:
https://jaeger.dev.storj.io/trace/4df8f6538811f766
FAIL (2026-03-18 20:13:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Mar 18 20:13:14.354937904 (196h2m17.37060751s) [upload=188305]: Starting upload
Mar 18 20:13:14.355011049 (196h2m17.370680639s) [upload=188305]: Done starting upload
Mar 18 20:13:14.355299147 (196h2m17.370968736s) [upload=188305]: Uploading segments...
Mar 18 20:13:14.355370443 (196h2m17.371040054s) [upload=188305]: Got next segment. Inline: false
Mar 18 20:13:14.355409298 (196h2m17.371078901s) [upload=188305]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:13:14.355430832 (196h2m17.371100435s) [upload=188305]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:13:16.834958230 (196h2m19.850627831s) [seg_pos={0 0}, upload=188305]: Begin upload segment...
Mar 18 20:13:16.835023997 (196h2m19.850693597s) [seg_pos={0 0}, upload=188305]: Done begin upload segment.
Mar 18 20:13:16.835064763 (196h2m19.850734364s) [noise=true , address=45.91.116.7:4012 , piece_id=UQW7DYUETWXG5GKVYBKIP376YLS2YU2GULVS4UIJFIP7SIQGEB6Q, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.835621867 (196h2m19.85129147s) [noise=true , address=145.79.227.31:4803 , piece_id=TI7GIJYALNLLGSW7UMALOZQLPEPEALEBJYVJGJXYVAYNXGRDEHOA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.835848947 (196h2m19.851518554s) [noise=true , address=45.91.116.13:4188 , piece_id=Y4RTYNLI6TTDBJP5TDTNUP6AJBM7MGHK7A2XFI7SIUHX5B57BCTQ, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.836235001 (196h2m19.85190461s) [noise=true , address=77.90.31.10:4522 , piece_id=XE56P6BNXPFGPENHPNTB2BHXV42SX4HYSWDY3FOY5FNPKZQAYTJA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.836984877 (196h2m19.852654486s) [noise=true , address=77.90.31.12:4594 , piece_id=S6U7OMFLHLY2QC6IDUTATM2N7OVN2UXDBSHVDUZHU62YOSAUR26Q, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.838283498 (196h2m19.853953104s) [noise=true , address=145.79.227.44:4197 , piece_id=IVHM5W7JXKU7KDSRIPBMBHMQK3ELPQLPRMO2MZ2FKORG5RGYTWGA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.838349018 (196h2m19.854018621s) [noise=true , address=145.79.227.24:4585 , piece_id=TEPYUD4BHMCHFRP5ROENFFW6EC5SSKZS4OJ26UUZ72OG3UWUVNLA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.839075222 (196h2m19.85474483s) [noise=true , address=145.79.227.34:4910 , piece_id=QLSA5WE5XHOQ6VR6QLEOZUFX3ZLJPMGVG4JZXZ2OMV2C6HLUCFTA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.839266483 (196h2m19.854936085s) [noise=true , address=145.79.227.41:4087 , piece_id=NTPD2S4Q3XGXC4UN7KALS2S4POJ22FWDP3UZPTBT2Y65FESVX67Q, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.839925017 (196h2m19.855594616s) [noise=true , address=45.91.116.5:4332 , piece_id=IV2ANOHFT6ISMBMUIJXK32GHE27CD64R3NWTBFDPUEWX3I6S3HMQ, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.839972036 (196h2m19.855641642s) [noise=true , address=145.79.227.20:4441 , piece_id=WGHNTAQ35B4NMZJENGMKG6TJMJIRBBFDQPNFG5RJDTWAMA7FFQYQ, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.840590665 (196h2m19.856260271s) [noise=true , address=45.91.116.11:4118 , piece_id=S3AMOOH6ANSNN4JTWZV5N2AHJ756SBFOYIGJ6EIYQOXNYYVQ2MCA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.840800863 (196h2m19.856470466s) [noise=true , address=45.91.116.8:4046 , piece_id=KO7KB47XGAEUEGAYLAAL7AJMZSJMMDMC67AAIHUEKKLMGSCYOGGA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.843419599 (196h2m19.859089194s) [noise=true , address=145.79.227.32:4837 , piece_id=MLXDI3OPZLO465PF3V4OJYKWQZLJFLR7IBE6UCG2UNTJWWBQU22Q, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.844229690 (196h2m19.859899283s) [noise=true , address=45.91.116.10:4085 , piece_id=JM35UZHZYJYCRO66IULRQNKUVGM6MFDVT73B7X2BQ6XKWL4ETMVA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:16.844937348 (196h2m19.860606935s) [noise=true , address=45.91.116.6:4989 , piece_id=U6XHKJNRWJQ7FXP3NYHS4LIQJFMXKR7MTQOH2Y5R6L3UUUJQC3ZA, seg_pos={0 0}, upload=188305]: Uploading piece...
Mar 18 20:13:17.434373901 (196h2m20.450043491s) [upload=188305]: Next returned nil segment
Mar 18 20:13:17.434389632 (196h2m20.450059241s) [upload=188305]: Waiting for error group managing segments...
Mar 18 20:13:17.575633989 (196h2m20.591303601s) [noise=true , address=45.91.116.7:4012 , piece_id=UQW7DYUETWXG5GKVYBKIP376YLS2YU2GULVS4UIJFIP7SIQGEB6Q, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.581533247 (196h2m20.597202858s) [noise=true , address=45.91.116.11:4118 , piece_id=S3AMOOH6ANSNN4JTWZV5N2AHJ756SBFOYIGJ6EIYQOXNYYVQ2MCA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.582667981 (196h2m20.598337591s) [noise=true , address=77.90.31.10:4522 , piece_id=XE56P6BNXPFGPENHPNTB2BHXV42SX4HYSWDY3FOY5FNPKZQAYTJA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.584090464 (196h2m20.599760072s) [noise=true , address=45.91.116.5:4332 , piece_id=IV2ANOHFT6ISMBMUIJXK32GHE27CD64R3NWTBFDPUEWX3I6S3HMQ, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.584745231 (196h2m20.600414839s) [noise=true , address=45.91.116.10:4085 , piece_id=JM35UZHZYJYCRO66IULRQNKUVGM6MFDVT73B7X2BQ6XKWL4ETMVA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.588416935 (196h2m20.604086541s) [noise=true , address=45.91.116.6:4989 , piece_id=U6XHKJNRWJQ7FXP3NYHS4LIQJFMXKR7MTQOH2Y5R6L3UUUJQC3ZA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.595781267 (196h2m20.61145087s) [noise=true , address=45.91.116.13:4188 , piece_id=Y4RTYNLI6TTDBJP5TDTNUP6AJBM7MGHK7A2XFI7SIUHX5B57BCTQ, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.605473159 (196h2m20.621142768s) [noise=true , address=45.91.116.8:4046 , piece_id=KO7KB47XGAEUEGAYLAAL7AJMZSJMMDMC67AAIHUEKKLMGSCYOGGA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:17.609297617 (196h2m20.624967223s) [noise=true , address=77.90.31.12:4594 , piece_id=S6U7OMFLHLY2QC6IDUTATM2N7OVN2UXDBSHVDUZHU62YOSAUR26Q, seg_pos={0 0}, upload=188305]: Done uploading piece. err: <nil>
Mar 18 20:13:44.054807254 (196h2m47.070476862s) [noise=true , address=145.79.227.24:4585 , piece_id=TEPYUD4BHMCHFRP5ROENFFW6EC5SSKZS4OJ26UUZ72OG3UWUVNLA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: rpc: dial tcp 145.79.227.24:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.054871399 (196h2m47.070541008s) [noise=true , address=145.79.227.32:4837 , piece_id=MLXDI3OPZLO465PF3V4OJYKWQZLJFLR7IBE6UCG2UNTJWWBQU22Q, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:1jmioLYNGZjDcByx8B1oHbqfid2DRqRfgoHhz85VcoTqbWz6Ls, address:145.79.227.32:4837): protocol: rpc: dial tcp 145.79.227.32:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.054881694 (196h2m47.070551288s) [noise=true , address=145.79.227.34:4910 , piece_id=QLSA5WE5XHOQ6VR6QLEOZUFX3ZLJPMGVG4JZXZ2OMV2C6HLUCFTA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:12bRYzNggkq7QWFTP9fB94dTrMMX8AYvKMt2zjmJBzT1VwqyPVZ, address:145.79.227.34:4910): protocol: rpc: dial tcp 145.79.227.34:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.054927602 (196h2m47.070597212s) [noise=true , address=145.79.227.41:4087 , piece_id=NTPD2S4Q3XGXC4UN7KALS2S4POJ22FWDP3UZPTBT2Y65FESVX67Q, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: rpc: dial tcp 145.79.227.41:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.054966119 (196h2m47.070635734s) [noise=true , address=145.79.227.44:4197 , piece_id=IVHM5W7JXKU7KDSRIPBMBHMQK3ELPQLPRMO2MZ2FKORG5RGYTWGA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:145.79.227.44:4197): protocol: rpc: dial tcp 145.79.227.44:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.055032355 (196h2m47.070701967s) [noise=true , address=145.79.227.20:4441 , piece_id=WGHNTAQ35B4NMZJENGMKG6TJMJIRBBFDQPNFG5RJDTWAMA7FFQYQ, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: rpc: dial tcp 145.79.227.20:4441: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.055083370 (196h2m47.070752971s) [noise=true , address=145.79.227.31:4803 , piece_id=TI7GIJYALNLLGSW7UMALOZQLPEPEALEBJYVJGJXYVAYNXGRDEHOA, seg_pos={0 0}, upload=188305]: Done uploading piece. err: ecclient: upload failed (node:12GFk3GkhB88UKQdZM8vZ9VLNuuYDrS5ThZjVAdbL2CtTQVMDEB, address:145.79.227.31:4803): protocol: rpc: dial tcp 145.79.227.31:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:13:44.055107374 (196h2m47.070776974s) [seg_pos={0 0}, upload=188305]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Mar 18 20:13:44.168234368 (196h2m47.183903969s) [upload=188305]: Done uploading segments...
Mar 18 20:13:44.168271973 (196h2m47.183941579s) [upload=188305]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.11556986s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.............XXX.XXX...............X........................] duration:
18.91578119s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:13:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.91581271s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
49 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.110385287s trace:
https://jaeger.dev.storj.io/trace/5e32a429a62026f3
FAIL (2026-03-18 20:13:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:12:33.569771310 (196h1m36.585440921s) [upload=188294]: Starting upload
Mar 18 20:12:33.569891777 (196h1m36.585561384s) [upload=188294]: Done starting upload
Mar 18 20:12:33.570009655 (196h1m36.585679257s) [upload=188294]: Uploading segments...
Mar 18 20:12:33.570558461 (196h1m36.586228071s) [upload=188294]: Got next segment. Inline: false
Mar 18 20:12:33.570605050 (196h1m36.586274661s) [upload=188294]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:12:33.570610985 (196h1m36.586280593s) [upload=188294]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:12:37.144132897 (196h1m40.159802507s) [seg_pos={0 0}, upload=188294]: Begin upload segment...
Mar 18 20:12:37.144197287 (196h1m40.15986689s) [seg_pos={0 0}, upload=188294]: Done begin upload segment.
Mar 18 20:12:37.144254589 (196h1m40.159924189s) [noise=true , address=77.90.31.14:4661 , piece_id=WDQ7OMHKWIN73SGPGQVDMWRYY7ZZZCAQBHMY42VLOXATB5T74KCA, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.144267132 (196h1m40.159936736s) [noise=true , address=77.90.31.13:4630 , piece_id=JQMHRKF2IWHGRGY4F5OEJI25QU4ZCJUTSOXKUDWPETHGA63RG4ZA, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.144338503 (196h1m40.160008114s) [noise=true , address=45.91.116.9:4265 , piece_id=77CZP2MSDGWJ7POUOQT4RYFHCN3KARHJLG7MHT3DPYEBLQEJ7NWQ, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.144879922 (196h1m40.160549534s) [noise=true , address=77.90.31.19:4844 , piece_id=FRVTN7GAVI6IVO3SJ6MN5BYG3KRDUGLJ7UAD54AIBPMODLMCLGUA, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.144891550 (196h1m40.160561154s) [noise=true , address=145.79.227.29:4944 , piece_id=7LRKJ2CPJRLSR7N26TLOTFNM2CN6PECB7QWS52WBUKF46U7W2JBA, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.145322063 (196h1m40.160991674s) [noise=true , address=145.79.227.37:4567 , piece_id=L56KB5BSRI4VO7U4JITIBCCIWI2QQS4USWHGY7HOSXILQNERURSQ, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.145377183 (196h1m40.161046793s) [noise=true , address=79.127.219.41:4928 , piece_id=52MV4XIZ6ZP4CLN4MD4NA3AMYH7QHVGPUD4ALOWCIG46B4SJPKIA, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.145855764 (196h1m40.161525372s) [noise=true , address=145.79.227.28:4729 , piece_id=YCQQRIALAW56YVLHSMJZY4I2O3R54CCWDD2RUEFGQI7MNQCIYJ7A, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.146059523 (196h1m40.16172913s) [noise=true , address=77.90.31.12:4593 , piece_id=7UT6ZGDS6XSVZOHZE3USG5MKAHL5M54CAO4GSTETP6CWX4OPILBA, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.146328221 (196h1m40.161997831s) [noise=true , address=45.91.116.8:4049 , piece_id=LXBBPOTT2Y36KP3SFA4KR4CPK2M7GBKW3UX5XH2VTEAGUROKVU2A, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.146545743 (196h1m40.162215348s) [noise=true , address=45.91.116.13:4157 , piece_id=2STCNRLDRUAEZW55RTUXM73YMMONWSKMB2VAABAUIKOF7C4AF2PQ, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.146717392 (196h1m40.162387002s) [noise=true , address=45.91.116.6:4989 , piece_id=MKTQ6L6XMIMMWAQ6HKZCGB6R5TVMRUVFQ54CFXGQ5YNDROZML5HQ, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.147042989 (196h1m40.162712594s) [noise=true , address=145.79.227.22:4515 , piece_id=EP5B67BI3I5L6M45KEZ6R6E2C7RHQ2NWJ3UGWYUZS5IFE53LR6EQ, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.147117443 (196h1m40.162787031s) [noise=true , address=77.90.31.11:4556 , piece_id=ZDKQI4CQGIG4WTDWS7G4SCQTNCUNM7HLZESZWIHT6PKNWBP6E3NQ, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.149492820 (196h1m40.165162429s) [noise=true , address=45.91.116.11:4121 , piece_id=HOAVSVA7G5XJPPPUQA2IVKFY7OTBNLHYBWBCZN2BNRNNNTK5FFHQ, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.150613916 (196h1m40.166283516s) [noise=true , address=45.91.116.4:4953 , piece_id=XCGCAZSRKO2BEPRNIEENWSP7WF36RRMGFBVXPWXNMKGTPZQ3HDKQ, seg_pos={0 0}, upload=188294]: Uploading piece...
Mar 18 20:12:37.747330348 (196h1m40.762999948s) [upload=188294]: Next returned nil segment
Mar 18 20:12:37.747343563 (196h1m40.763013166s) [upload=188294]: Waiting for error group managing segments...
Mar 18 20:12:37.879717105 (196h1m40.895386706s) [noise=true , address=77.90.31.13:4630 , piece_id=JQMHRKF2IWHGRGY4F5OEJI25QU4ZCJUTSOXKUDWPETHGA63RG4ZA, seg_pos={0 0}, upload=188294]: Done uploading piece. err: <nil>
Mar 18 20:12:37.886779812 (196h1m40.902449405s) [noise=true , address=45.91.116.4:4953 , piece_id=XCGCAZSRKO2BEPRNIEENWSP7WF36RRMGFBVXPWXNMKGTPZQ3HDKQ, seg_pos={0 0}, upload=188294]: Done uploading piece. err: <nil>
Mar 18 20:12:37.887852343 (196h1m40.903521974s) [noise=true , address=77.90.31.11:4556 , piece_id=ZDKQI4CQGIG4WTDWS7G4SCQTNCUNM7HLZESZWIHT6PKNWBP6E3NQ, seg_pos={0 0}, upload=188294]: Done uploading piece. err: <nil>
Mar 18 20:12:37.895763188 (196h1m40.911432778s) [noise=true , address=45.91.116.11:4121 , piece_id=HOAVSVA7G5XJPPPUQA2IVKFY7OTBNLHYBWBCZN2BNRNNNTK5FFHQ, seg_pos={0 0}, upload=188294]: Done uploading piece. err: <nil>
Mar 18 20:12:37.897881419 (196h1m40.913551024s) [noise=true , address=77.90.31.19:4844 , piece_id=FRVTN7GAVI6IVO3SJ6MN5BYG3KRDUGLJ7UAD54AIBPMODLMCLGUA, seg_pos={0 0}, upload=188294]: Done uploading piece. err: <nil>
Mar 18 20:12:37.898992174 (196h1m40.914661763s) [noise=true , address=45.91.116.13:4157 , piece_id=2STCNRLDRUAEZW55RTUXM73YMMONWSKMB2VAABAUIKOF7C4AF2PQ, seg_pos={0 0}, upload=188294]: Done uploading piece. err: <nil>
Mar 18 20:12:37.899729695 (196h1m40.9153993s) [noise=true , address=79.127.219.41:4928 , piece_id=52MV4XIZ6ZP4CLN4MD4NA3AMYH7QHVGPUD4ALOWCIG46B4SJPKIA, seg_pos={0 0}, upload=188294]: Done uploading piece. err: <nil>
Mar 18 20:12:37.901540439 (196h1m40.917210046s) [noise=true , address=45.91.116.9:4265 , piece_id=77CZP2MSDGWJ7POUOQT4RYFHCN3KARHJLG7MHT3DPYEBLQEJ7NWQ, seg_pos={0 0}, upload=188294]: Done uploading piece. err: <nil>
Mar 18 20:12:37.904012682 (196h1m40.919682283s) [noise=true , address=45.91.116.6:4989 , piece_id=MKTQ6L6XMIMMWAQ6HKZCGB6R5TVMRUVFQ54CFXGQ5YNDROZML5HQ, seg_pos={0 0}, upload=188294]: Done uploading piece. err: <nil>
Mar 18 20:12:37.904834130 (196h1m40.92050373s) [noise=true , address=45.91.116.8:4049 , piece_id=LXBBPOTT2Y36KP3SFA4KR4CPK2M7GBKW3UX5XH2VTEAGUROKVU2A, seg_pos={0 0}, upload=188294]: Done uploading piece. err: <nil>
Mar 18 20:12:37.906968609 (196h1m40.922638226s) [noise=true , address=77.90.31.12:4593 , piece_id=7UT6ZGDS6XSVZOHZE3USG5MKAHL5M54CAO4GSTETP6CWX4OPILBA, seg_pos={0 0}, upload=188294]: Done uploading piece. err: <nil>
Mar 18 20:12:37.920120397 (196h1m40.935790001s) [noise=true , address=77.90.31.14:4661 , piece_id=WDQ7OMHKWIN73SGPGQVDMWRYY7ZZZCAQBHMY42VLOXATB5T74KCA, seg_pos={0 0}, upload=188294]: Done uploading piece. err: <nil>
Mar 18 20:13:03.285300844 (196h2m6.300970453s) [noise=true , address=145.79.227.37:4567 , piece_id=L56KB5BSRI4VO7U4JITIBCCIWI2QQS4USWHGY7HOSXILQNERURSQ, seg_pos={0 0}, upload=188294]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: rpc: dial tcp 145.79.227.37:4567: i/o timeout; conn closed: op
Mar 18 20:13:03.285426794 (196h2m6.301096407s) [noise=true , address=145.79.227.22:4515 , piece_id=EP5B67BI3I5L6M45KEZ6R6E2C7RHQ2NWJ3UGWYUZS5IFE53LR6EQ, seg_pos={0 0}, upload=188294]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:13:03.285761512 (196h2m6.301431124s) [noise=true , address=145.79.227.29:4944 , piece_id=7LRKJ2CPJRLSR7N26TLOTFNM2CN6PECB7QWS52WBUKF46U7W2JBA, seg_pos={0 0}, upload=188294]: Done uploading piece. err: ecclient: upload failed (node:12KMFsxzp7SfMiFBuMTaWGVyJoRTMw7ZRcfEpPDv1BmbViGF6K9, address:145.79.227.29:4944): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:13:03.286007913 (196h2m6.301677523s) [noise=true , address=145.79.227.28:4729 , piece_id=YCQQRIALAW56YVLHSMJZY4I2O3R54CCWDD2RUEFGQI7MNQCIYJ7A, seg_pos={0 0}, upload=188294]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:13:03.286046880 (196h2m6.301716468s) [seg_pos={0 0}, upload=188294]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:13:03.393308031 (196h2m6.408977642s) [upload=188294]: Done uploading segments...
Mar 18 20:13:03.393360173 (196h2m6.409029778s) [upload=188294]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.11040242s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.............X..X...X..X..X........X..X..X...X..............] duration:
19.850952186s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:12:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 19.850982974s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [.......XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001581184s trace:
https://jaeger.dev.storj.io/trace/648798cc2b658a35
FAIL (2026-03-18 20:12:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001605722s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
48 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration:
30.115428534s trace:
https://jaeger.dev.storj.io/trace/59dcbb6c7634bbd0
FAIL (2026-03-18 20:12:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Mar 18 20:12:03.291890263 (196h1m6.307559863s) [upload=188286]: Starting upload
Mar 18 20:12:03.291967454 (196h1m6.30763706s) [upload=188286]: Done starting upload
Mar 18 20:12:03.292069060 (196h1m6.307738651s) [upload=188286]: Uploading segments...
Mar 18 20:12:03.292335772 (196h1m6.308005372s) [upload=188286]: Got next segment. Inline: false
Mar 18 20:12:03.292394045 (196h1m6.308063632s) [upload=188286]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:12:03.292399988 (196h1m6.308069579s) [upload=188286]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:12:06.519009722 (196h1m9.534679331s) [seg_pos={0 0}, upload=188286]: Begin upload segment...
Mar 18 20:12:06.519084965 (196h1m9.534754551s) [seg_pos={0 0}, upload=188286]: Done begin upload segment.
Mar 18 20:12:06.519122012 (196h1m9.534791626s) [noise=true , address=79.127.219.45:4855 , piece_id=36YTTFA5J36UBYYYFPF2DGCVVL5C7DND5CZF57LMJNPBBIE3KNPA, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.522222666 (196h1m9.537892272s) [noise=true , address=77.90.31.18:4808 , piece_id=BZBWITQBW6JD6AETB5U4QBDVU3O4LVN67OB2CKZAEFJI4H5MPFBQ, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.522581285 (196h1m9.538250895s) [noise=true , address=145.79.227.29:4945 , piece_id=CYKKASYWHV4WMBYPL3DG5BDRB4I4INJ2MXZFHSSBICMUXVH6LXDA, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.523159155 (196h1m9.538828755s) [noise=true , address=45.91.116.6:4989 , piece_id=7OGSM7WHMXSM76JPXWNEIRVRWFSTHWSC7RCGWCLNEKJAEHBQ4G7Q, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.523628162 (196h1m9.539297787s) [noise=true , address=145.79.227.33:4870 , piece_id=Z4KXLCM24LES4M2FEAZ64EHY2IISSTM6VY64GTYKUVAETVMEWCRA, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.523675502 (196h1m9.539345117s) [noise=true , address=45.91.116.9:4295 , piece_id=YQ323FZE2YHJMZMT6GHLJPMGNBOCPYS63LXC2TJEBMU3AYRHWK3A, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.524128377 (196h1m9.539798035s) [noise=true , address=45.91.116.14:4191 , piece_id=OKEGIIY74BDSVMDAWEJQVPYXFXFYXIYLNKFT45GCWSBZPP72O7YQ, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.524219590 (196h1m9.539889202s) [noise=true , address=45.91.116.8:4080 , piece_id=WYUMVYWWFPO57LC22APTIEK724U4C4UOTARZ6D4QW4QIJ6PGQSGA, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.524539587 (196h1m9.540209187s) [noise=true , address=45.91.116.7:4024 , piece_id=AT23FNKZX64YAUZLCI5S2FYHRZKKZA6SFELEE3V2SQYJSBYWI6IQ, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.524848506 (196h1m9.540518117s) [noise=true , address=145.79.227.21:4479 , piece_id=KKQNF2TAHLQ2WYFPFDE4EH3P37OQ7GWJKPNTY5H5PRSR2VXDHSDQ, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.524979019 (196h1m9.540648619s) [noise=true , address=79.127.219.46:4889 , piece_id=4RVRPW6LAOB3UATZJFE46VMQ63CDLIY3IYUZWBSQBM76XIZPCMCA, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.525382533 (196h1m9.541052133s) [noise=true , address=145.79.227.26:4659 , piece_id=RHGXZ4VL66KTUI73EV56A3SKZWR5TWSWIB2JODBXN4PFNRHZPKCQ, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.525604109 (196h1m9.541273719s) [noise=true , address=145.79.227.27:4694 , piece_id=LQWE3QTUOONYI4REAOWSFVKJFDLYPN7RELCYIDUOVAZVJ3SSVYNA, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.526519516 (196h1m9.542189126s) [noise=true , address=145.79.227.42:4125 , piece_id=KNPMMNFOUWEC63RUTLJ53CKLVJXU3D254YVRTNUVM3SCJMJ736UA, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.527068837 (196h1m9.54273844s) [noise=true , address=77.90.31.17:4770 , piece_id=GK642OOLZ22IKR7LBFU5LFWHDQNEP6TDHRU7U2ZPUG6VNNLDQSEQ, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:06.527738202 (196h1m9.543407805s) [noise=true , address=145.79.227.7:4295 , piece_id=C5E47ZHDDHVSTRRTYUNBKLH7D6UQVYUSIUBFWBKNIO4CKYPJJ2KQ, seg_pos={0 0}, upload=188286]: Uploading piece...
Mar 18 20:12:07.129004272 (196h1m10.14467386s) [upload=188286]: Next returned nil segment
Mar 18 20:12:07.129024893 (196h1m10.144694505s) [upload=188286]: Waiting for error group managing segments...
Mar 18 20:12:07.251314266 (196h1m10.26698387s) [noise=true , address=77.90.31.18:4808 , piece_id=BZBWITQBW6JD6AETB5U4QBDVU3O4LVN67OB2CKZAEFJI4H5MPFBQ, seg_pos={0 0}, upload=188286]: Done uploading piece. err: <nil>
Mar 18 20:12:07.269376139 (196h1m10.28504574s) [noise=true , address=45.91.116.14:4191 , piece_id=OKEGIIY74BDSVMDAWEJQVPYXFXFYXIYLNKFT45GCWSBZPP72O7YQ, seg_pos={0 0}, upload=188286]: Done uploading piece. err: <nil>
Mar 18 20:12:07.270785480 (196h1m10.286455087s) [noise=true , address=77.90.31.17:4770 , piece_id=GK642OOLZ22IKR7LBFU5LFWHDQNEP6TDHRU7U2ZPUG6VNNLDQSEQ, seg_pos={0 0}, upload=188286]: Done uploading piece. err: <nil>
Mar 18 20:12:07.271270435 (196h1m10.286940036s) [noise=true , address=45.91.116.7:4024 , piece_id=AT23FNKZX64YAUZLCI5S2FYHRZKKZA6SFELEE3V2SQYJSBYWI6IQ, seg_pos={0 0}, upload=188286]: Done uploading piece. err: <nil>
Mar 18 20:12:07.278716081 (196h1m10.294385684s) [noise=true , address=45.91.116.6:4989 , piece_id=7OGSM7WHMXSM76JPXWNEIRVRWFSTHWSC7RCGWCLNEKJAEHBQ4G7Q, seg_pos={0 0}, upload=188286]: Done uploading piece. err: <nil>
Mar 18 20:12:07.279751551 (196h1m10.295421152s) [noise=true , address=45.91.116.8:4080 , piece_id=WYUMVYWWFPO57LC22APTIEK724U4C4UOTARZ6D4QW4QIJ6PGQSGA, seg_pos={0 0}, upload=188286]: Done uploading piece. err: <nil>
Mar 18 20:12:07.286750966 (196h1m10.302420567s) [noise=true , address=45.91.116.9:4295 , piece_id=YQ323FZE2YHJMZMT6GHLJPMGNBOCPYS63LXC2TJEBMU3AYRHWK3A, seg_pos={0 0}, upload=188286]: Done uploading piece. err: <nil>
Mar 18 20:12:07.314790937 (196h1m10.330460527s) [noise=true , address=79.127.219.46:4889 , piece_id=4RVRPW6LAOB3UATZJFE46VMQ63CDLIY3IYUZWBSQBM76XIZPCMCA, seg_pos={0 0}, upload=188286]: Done uploading piece. err: <nil>
Mar 18 20:12:07.335737687 (196h1m10.351407293s) [noise=true , address=79.127.219.45:4855 , piece_id=36YTTFA5J36UBYYYFPF2DGCVVL5C7DND5CZF57LMJNPBBIE3KNPA, seg_pos={0 0}, upload=188286]: Done uploading piece. err: <nil>
Mar 18 20:12:33.006808040 (196h1m36.022477651s) [noise=true , address=145.79.227.21:4479 , piece_id=KKQNF2TAHLQ2WYFPFDE4EH3P37OQ7GWJKPNTY5H5PRSR2VXDHSDQ, seg_pos={0 0}, upload=188286]: Done uploading piece. err: ecclient: upload failed (node:12VCopFEoHk99GsQNQPuWvTQZRD1bpXi8HHLKXaXRdvFThTxmAk, address:145.79.227.21:4479): protocol: rpc: dial tcp 145.79.227.21:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:12:33.007248536 (196h1m36.022918158s) [noise=true , address=145.79.227.26:4659 , piece_id=RHGXZ4VL66KTUI73EV56A3SKZWR5TWSWIB2JODBXN4PFNRHZPKCQ, seg_pos={0 0}, upload=188286]: Done uploading piece. err: ecclient: upload failed (node:12UBuWLogtN8mv4nwXDpzGdQtbKDz1EfVENSjXC54QdFp2fJg99, address:145.79.227.26:4659): protocol: rpc: dial tcp 145.79.227.26:4659: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:12:33.007366218 (196h1m36.023035812s) [noise=true , address=145.79.227.7:4295 , piece_id=C5E47ZHDDHVSTRRTYUNBKLH7D6UQVYUSIUBFWBKNIO4CKYPJJ2KQ, seg_pos={0 0}, upload=188286]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:12:33.007550067 (196h1m36.023219679s) [noise=true , address=145.79.227.42:4125 , piece_id=KNPMMNFOUWEC63RUTLJ53CKLVJXU3D254YVRTNUVM3SCJMJ736UA, seg_pos={0 0}, upload=188286]: Done uploading piece. err: ecclient: upload failed (node:1MrdHwiDVe8m6wcaQJrYetza9qckLruJXiU9UYbMxGCSWWMV1S, address:145.79.227.42:4125): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:12:33.007637753 (196h1m36.023307363s) [noise=true , address=145.79.227.29:4945 , piece_id=CYKKASYWHV4WMBYPL3DG5BDRB4I4INJ2MXZFHSSBICMUXVH6LXDA, seg_pos={0 0}, upload=188286]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: rpc: dial tcp 145.79.227.29:4945: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:12:33.008023293 (196h1m36.023692895s) [noise=true , address=145.79.227.33:4870 , piece_id=Z4KXLCM24LES4M2FEAZ64EHY2IISSTM6VY64GTYKUVAETVMEWCRA, seg_pos={0 0}, upload=188286]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: rpc: dial tcp 145.79.227.33:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:12:33.008166722 (196h1m36.023836331s) [noise=true , address=145.79.227.27:4694 , piece_id=LQWE3QTUOONYI4REAOWSFVKJFDLYPN7RELCYIDUOVAZVJ3SSVYNA, seg_pos={0 0}, upload=188286]: Done uploading piece. err: ecclient: upload failed (node:17sHakGWopKRax7hbzezZASaWtH5tUUdFMP5diZDK3UwP153cd, address:145.79.227.27:4694): protocol: rpc: dial tcp 145.79.227.27:4694: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:12:33.008198639 (196h1m36.023868228s) [seg_pos={0 0}, upload=188286]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Mar 18 20:12:33.119927152 (196h1m36.135596752s) [upload=188286]: Done uploading segments...
Mar 18 20:12:33.119975929 (196h1m36.135645529s) [upload=188286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.115473032s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [..............XXX.XXX...............X.......................] duration:
19.494121569s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:12:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.494152981s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
47 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration:
30.11002755s trace:
https://jaeger.dev.storj.io/trace/65579914ddbd4280
FAIL (2026-03-18 20:12:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Mar 18 20:11:30.992759148 (196h0m34.008428758s) [upload=188277]: Starting upload
Mar 18 20:11:30.992978833 (196h0m34.008648434s) [upload=188277]: Done starting upload
Mar 18 20:11:30.993098409 (196h0m34.008768009s) [upload=188277]: Uploading segments...
Mar 18 20:11:30.993427918 (196h0m34.009097507s) [upload=188277]: Got next segment. Inline: false
Mar 18 20:11:30.993725913 (196h0m34.009395515s) [upload=188277]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:11:30.993735519 (196h0m34.009405122s) [upload=188277]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:11:34.769007649 (196h0m37.784677259s) [seg_pos={0 0}, upload=188277]: Begin upload segment...
Mar 18 20:11:34.769085350 (196h0m37.784754952s) [seg_pos={0 0}, upload=188277]: Done begin upload segment.
Mar 18 20:11:34.769128336 (196h0m37.784797939s) [noise=true , address=145.79.227.41:4087 , piece_id=HUVHKFQM6J7F5ARHP6LBDLFZR2DWLPKAS7TYEJFXFQQGUGKY24MA, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.770234105 (196h0m37.785903702s) [noise=true , address=77.90.31.11:4557 , piece_id=5NBFXP23GI5FT5QVOOBZTCR7S65RQSINEFLCT3WYJ34UPELZQ5BA, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.770350087 (196h0m37.786019709s) [noise=true , address=77.90.31.17:4774 , piece_id=MLOMJHYUWQTAH2NI4G6FR3MOWJF6CCZMWZPGJYMAZAWGVV7W2KYQ, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.770829062 (196h0m37.786498662s) [noise=true , address=145.79.227.6:4261 , piece_id=LT3H6UEMLSL43L4RE6RQA5LDSOVGAU5EGLYBXCGRXNXYFXRYZ7YA, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.771006433 (196h0m37.786676042s) [noise=true , address=145.79.227.37:4569 , piece_id=3IUENIY6DWAVRS75IK6ZER7AM5DNFX7G7DEJEGTEXWCPBM6SVWYQ, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.771318719 (196h0m37.786988322s) [noise=true , address=45.91.116.7:4019 , piece_id=P4YPBCNEGMNK7NK3ZOBPCHE4DT2B3GORANUY4WXWT5QCN6OY4OHA, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.771494577 (196h0m37.787164188s) [noise=true , address=145.79.227.20:4441 , piece_id=ATPIGBTV3KTGRQRE4FV4SFKH337NPIF4UN4KARGTO36WAYMTTWRQ, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.771730393 (196h0m37.78740001s) [noise=true , address=145.79.227.30:4765 , piece_id=DBS45ZRHQ3IOYJW7YCIJW4F52YZBOY2PAWZWRO6OKFBRSUENMJ7Q, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.771958679 (196h0m37.787628291s) [noise=true , address=45.91.116.5:4297 , piece_id=HWMCDRRHMPJARKMM722GS6QRQYVBUMMWFP5AQCSY3XYJNIR3OVJQ, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.772205396 (196h0m37.787874999s) [noise=true , address=77.90.31.21:4880 , piece_id=JXD7DHT3GGUZZANIGXCFIAH7J3QV5JSOE44KWD5ZW4AKOYO2UPDQ, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.772379191 (196h0m37.788048802s) [noise=true , address=45.91.116.14:4216 , piece_id=KXXE7RFFDV3AQNQNGFU25RIZB44GM5BSQ7P6U6VLKH2DVRJ7D42A, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.772780762 (196h0m37.788450415s) [noise=true , address=77.90.31.18:4808 , piece_id=D2YYQJYBICUGMGI7RSPO6R57Z5D3N6RIDOAACFU2UXADYNGZZTYQ, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.772845600 (196h0m37.788515207s) [noise=true , address=145.79.227.33:4871 , piece_id=H4RDJHKDBJJWAETXPQM4RBMYNQX7ENS2NQFB4BMI37WE2ILIV54Q, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.773304988 (196h0m37.788974599s) [noise=true , address=45.91.116.4:4953 , piece_id=WIIRPDA6K5O46D42PZIZ5ZMRKWHHLIAEFYH4LWALLHBAFW57RGMQ, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.773606050 (196h0m37.789275665s) [noise=true , address=45.91.116.10:4083 , piece_id=SHLEIKP5U6PHPVWRLYMJYIHKLS63JNY4RJA5XTR3HXM7GSKTW5DQ, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:34.773867362 (196h0m37.789536972s) [noise=true , address=79.127.219.45:4853 , piece_id=HKKCTXWVMO4DIF3YWSVRAN7QVDX6J4KDU2OUMOYIZGOCN4Y3CKAQ, seg_pos={0 0}, upload=188277]: Uploading piece...
Mar 18 20:11:35.371726089 (196h0m38.38739569s) [upload=188277]: Next returned nil segment
Mar 18 20:11:35.371741593 (196h0m38.387411194s) [upload=188277]: Waiting for error group managing segments...
Mar 18 20:11:35.506443866 (196h0m38.522113467s) [noise=true , address=77.90.31.21:4880 , piece_id=JXD7DHT3GGUZZANIGXCFIAH7J3QV5JSOE44KWD5ZW4AKOYO2UPDQ, seg_pos={0 0}, upload=188277]: Done uploading piece. err: <nil>
Mar 18 20:11:35.508788266 (196h0m38.524457877s) [noise=true , address=77.90.31.17:4774 , piece_id=MLOMJHYUWQTAH2NI4G6FR3MOWJF6CCZMWZPGJYMAZAWGVV7W2KYQ, seg_pos={0 0}, upload=188277]: Done uploading piece. err: <nil>
Mar 18 20:11:35.512885402 (196h0m38.528555009s) [noise=true , address=45.91.116.14:4216 , piece_id=KXXE7RFFDV3AQNQNGFU25RIZB44GM5BSQ7P6U6VLKH2DVRJ7D42A, seg_pos={0 0}, upload=188277]: Done uploading piece. err: <nil>
Mar 18 20:11:35.515155644 (196h0m38.530825244s) [noise=true , address=77.90.31.18:4808 , piece_id=D2YYQJYBICUGMGI7RSPO6R57Z5D3N6RIDOAACFU2UXADYNGZZTYQ, seg_pos={0 0}, upload=188277]: Done uploading piece. err: <nil>
Mar 18 20:11:35.515656175 (196h0m38.531325774s) [noise=true , address=45.91.116.10:4083 , piece_id=SHLEIKP5U6PHPVWRLYMJYIHKLS63JNY4RJA5XTR3HXM7GSKTW5DQ, seg_pos={0 0}, upload=188277]: Done uploading piece. err: <nil>
Mar 18 20:11:35.517218967 (196h0m38.532888568s) [noise=true , address=45.91.116.7:4019 , piece_id=P4YPBCNEGMNK7NK3ZOBPCHE4DT2B3GORANUY4WXWT5QCN6OY4OHA, seg_pos={0 0}, upload=188277]: Done uploading piece. err: <nil>
Mar 18 20:11:35.522533428 (196h0m38.538203027s) [noise=true , address=45.91.116.4:4953 , piece_id=WIIRPDA6K5O46D42PZIZ5ZMRKWHHLIAEFYH4LWALLHBAFW57RGMQ, seg_pos={0 0}, upload=188277]: Done uploading piece. err: <nil>
Mar 18 20:11:35.529202190 (196h0m38.544871793s) [noise=true , address=45.91.116.5:4297 , piece_id=HWMCDRRHMPJARKMM722GS6QRQYVBUMMWFP5AQCSY3XYJNIR3OVJQ, seg_pos={0 0}, upload=188277]: Done uploading piece. err: <nil>
Mar 18 20:11:35.532016549 (196h0m38.547686154s) [noise=true , address=77.90.31.11:4557 , piece_id=5NBFXP23GI5FT5QVOOBZTCR7S65RQSINEFLCT3WYJ34UPELZQ5BA, seg_pos={0 0}, upload=188277]: Done uploading piece. err: <nil>
Mar 18 20:11:35.574677760 (196h0m38.59034736s) [noise=true , address=79.127.219.45:4853 , piece_id=HKKCTXWVMO4DIF3YWSVRAN7QVDX6J4KDU2OUMOYIZGOCN4Y3CKAQ, seg_pos={0 0}, upload=188277]: Done uploading piece. err: <nil>
Mar 18 20:12:00.705352226 (196h1m3.721021867s) [noise=true , address=145.79.227.41:4087 , piece_id=HUVHKFQM6J7F5ARHP6LBDLFZR2DWLPKAS7TYEJFXFQQGUGKY24MA, seg_pos={0 0}, upload=188277]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: rpc: dial tcp 145.79.227.41:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:12:00.705826260 (196h1m3.721495869s) [noise=true , address=145.79.227.6:4261 , piece_id=LT3H6UEMLSL43L4RE6RQA5LDSOVGAU5EGLYBXCGRXNXYFXRYZ7YA, seg_pos={0 0}, upload=188277]: Done uploading piece. err: ecclient: upload failed (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz, address:145.79.227.6:4261): protocol: rpc: dial tcp 145.79.227.6:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:12:00.706069117 (196h1m3.721738747s) [noise=true , address=145.79.227.37:4569 , piece_id=3IUENIY6DWAVRS75IK6ZER7AM5DNFX7G7DEJEGTEXWCPBM6SVWYQ, seg_pos={0 0}, upload=188277]: Done uploading piece. err: ecclient: upload failed (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT, address:145.79.227.37:4569): protocol: rpc: dial tcp 145.79.227.37:4569: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:12:00.706123773 (196h1m3.721793383s) [noise=true , address=145.79.227.30:4765 , piece_id=DBS45ZRHQ3IOYJW7YCIJW4F52YZBOY2PAWZWRO6OKFBRSUENMJ7Q, seg_pos={0 0}, upload=188277]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: rpc: dial tcp 145.79.227.30:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:12:00.706465933 (196h1m3.72213554s) [noise=true , address=145.79.227.20:4441 , piece_id=ATPIGBTV3KTGRQRE4FV4SFKH337NPIF4UN4KARGTO36WAYMTTWRQ, seg_pos={0 0}, upload=188277]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: rpc: dial tcp 145.79.227.20:4441: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:12:00.706827901 (196h1m3.722497536s) [noise=true , address=145.79.227.33:4871 , piece_id=H4RDJHKDBJJWAETXPQM4RBMYNQX7ENS2NQFB4BMI37WE2ILIV54Q, seg_pos={0 0}, upload=188277]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:145.79.227.33:4871): protocol: rpc: dial tcp 145.79.227.33:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:12:00.706870869 (196h1m3.722540456s) [seg_pos={0 0}, upload=188277]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Mar 18 20:12:00.813180956 (196h1m3.828850565s) [upload=188277]: Done uploading segments...
Mar 18 20:12:00.813227432 (196h1m3.82889704s) [upload=188277]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.110053549s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [..............X..X...X..X..X........X..X..X...X.............] duration:
20.205011743s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:11:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 20.205034333s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [........XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000748385s trace:
https://jaeger.dev.storj.io/trace/4f61987f6d412ac8
FAIL (2026-03-18 20:11:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000770763s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [...............XXX.XXX...............X......................] duration:
19.265847083s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:11:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.265903706s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [.........XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.00070562s trace:
https://jaeger.dev.storj.io/trace/6b4a62ed823d0649
FAIL (2026-03-18 20:11:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000727442s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
46 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXX] duration:
30.112249966s trace:
https://jaeger.dev.storj.io/trace/6381f48cf69bdcaf
FAIL (2026-03-18 20:10:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Mar 18 20:10:20.921021301 (195h59m23.936690905s) [upload=188260]: Starting upload
Mar 18 20:10:20.921127811 (195h59m23.936797422s) [upload=188260]: Done starting upload
Mar 18 20:10:20.921374386 (195h59m23.937043986s) [upload=188260]: Uploading segments...
Mar 18 20:10:20.922037110 (195h59m23.937706718s) [upload=188260]: Got next segment. Inline: false
Mar 18 20:10:20.922128940 (195h59m23.937798545s) [upload=188260]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:10:20.922136186 (195h59m23.9378058s) [upload=188260]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:10:23.928080884 (195h59m26.943750484s) [seg_pos={0 0}, upload=188260]: Begin upload segment...
Mar 18 20:10:23.928146898 (195h59m26.943816489s) [seg_pos={0 0}, upload=188260]: Done begin upload segment.
Mar 18 20:10:23.928180354 (195h59m26.943849961s) [noise=true , address=45.91.116.11:4118 , piece_id=I5VRKSU7XQOYYKOXBHMUTAM44PIUHZ5YYZ2AZYPCKLVEITMIUIKA, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.928201440 (195h59m26.943871136s) [noise=true , address=145.79.227.21:4473 , piece_id=3EKSQJEBR2KC4OPW2SPTJPCD6CS7IMDRXW4TZ63XQPLPBWTEEDRA, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.928251703 (195h59m26.943921293s) [noise=true , address=45.91.116.12:4255 , piece_id=3237YNRW57FME6EYLESG2EDASXTHJPABOCYVMSK2BREJTQOOJPWA, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.928728716 (195h59m26.944398315s) [noise=true , address=79.127.219.45:4855 , piece_id=CU4IERVVHAUG6XXXUP6MHYSSDQMQX2Z3KWPP6QZSB6PCGHRC4UIQ, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.928748291 (195h59m26.944417897s) [noise=true , address=145.79.227.7:4295 , piece_id=DMCF33ZZFY4RXZMYKZWPC3DS2VZXCMFENRLPA6PAJHVL3W6Y2BSA, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.929261640 (195h59m26.944931243s) [noise=true , address=145.79.227.9:4368 , piece_id=JXSP3N56X2UDFQWGCXNPDMFKKJLK7HFU336Q7KUTE3H2R3I5TR5A, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.930156654 (195h59m26.945826257s) [noise=true , address=145.79.227.36:4521 , piece_id=MESIVF7I3PLOSK47LO4VMTMFPHN737AFCA5BQIQQRZPTS3YSK23A, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.930568078 (195h59m26.946237682s) [noise=true , address=145.79.227.42:4122 , piece_id=TR3IK2MPYDZXWII4PTFBBY2QNWBXQPOP4A7KRJS7CYK2V6EF3OTA, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.930948532 (195h59m26.946618138s) [noise=true , address=45.91.116.6:4989 , piece_id=FTTK7X3EOOXIPO64DMOIL47XTM5WQ6IUF6NPHBKRHLALP6CS5HHQ, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.931032553 (195h59m26.946702164s) [noise=true , address=77.90.31.11:4554 , piece_id=LIMZLLDMU7CBKWIUDJHBLUI2KFIXBY2V4XBD23UMPWQPQ2ZEF2CQ, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.931418007 (195h59m26.94708761s) [noise=true , address=45.91.116.4:4972 , piece_id=7Z7MFV747YQ4BBMJS7TZJEXTLEUUQNP5YR2GUJYMGQUGCBGDMOYA, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.931873895 (195h59m26.947543506s) [noise=true , address=45.91.116.9:4296 , piece_id=MGSTS5WUHCNGAUYVO4ZJ7D2OXYY5VYUHU7XRT5J3BOVUL3P3YK6Q, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.932039844 (195h59m26.947709453s) [noise=true , address=45.91.116.13:4157 , piece_id=UD6SW24UZINEIDS5IQBENER7Y4YEAZHZ26CTPF6TJ24UCB7XGF6Q, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.932454225 (195h59m26.948123832s) [noise=true , address=45.91.116.5:4301 , piece_id=HSBVXCAHRZRFTYYNLX7DBJ6FIJJLFRNSMJXFSHFBSXB7OQEBRCPA, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.932758874 (195h59m26.94842848s) [noise=true , address=145.79.227.43:4158 , piece_id=7WBRLGW753FYRJJTT4LPNGG622YBQUZDDTQZY37FRSY5ISK3UJSA, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:23.933300743 (195h59m26.948970346s) [noise=true , address=77.90.31.14:4662 , piece_id=LMB5VH2XB2ZQ5DEO3H5CHWSX4R2GDP4FNQFSFWY32LJ5KEYBRJPQ, seg_pos={0 0}, upload=188260]: Uploading piece...
Mar 18 20:10:24.529061836 (195h59m27.544731444s) [upload=188260]: Next returned nil segment
Mar 18 20:10:24.529078753 (195h59m27.544748361s) [upload=188260]: Waiting for error group managing segments...
Mar 18 20:10:24.667733508 (195h59m27.683403111s) [noise=true , address=45.91.116.4:4972 , piece_id=7Z7MFV747YQ4BBMJS7TZJEXTLEUUQNP5YR2GUJYMGQUGCBGDMOYA, seg_pos={0 0}, upload=188260]: Done uploading piece. err: <nil>
Mar 18 20:10:24.668587616 (195h59m27.684257217s) [noise=true , address=45.91.116.6:4989 , piece_id=FTTK7X3EOOXIPO64DMOIL47XTM5WQ6IUF6NPHBKRHLALP6CS5HHQ, seg_pos={0 0}, upload=188260]: Done uploading piece. err: <nil>
Mar 18 20:10:24.669358337 (195h59m27.685027937s) [noise=true , address=45.91.116.12:4255 , piece_id=3237YNRW57FME6EYLESG2EDASXTHJPABOCYVMSK2BREJTQOOJPWA, seg_pos={0 0}, upload=188260]: Done uploading piece. err: <nil>
Mar 18 20:10:24.670917823 (195h59m27.686587439s) [noise=true , address=45.91.116.11:4118 , piece_id=I5VRKSU7XQOYYKOXBHMUTAM44PIUHZ5YYZ2AZYPCKLVEITMIUIKA, seg_pos={0 0}, upload=188260]: Done uploading piece. err: <nil>
Mar 18 20:10:24.673385806 (195h59m27.689055406s) [noise=true , address=77.90.31.11:4554 , piece_id=LIMZLLDMU7CBKWIUDJHBLUI2KFIXBY2V4XBD23UMPWQPQ2ZEF2CQ, seg_pos={0 0}, upload=188260]: Done uploading piece. err: <nil>
Mar 18 20:10:24.677220345 (195h59m27.692889944s) [noise=true , address=45.91.116.9:4296 , piece_id=MGSTS5WUHCNGAUYVO4ZJ7D2OXYY5VYUHU7XRT5J3BOVUL3P3YK6Q, seg_pos={0 0}, upload=188260]: Done uploading piece. err: <nil>
Mar 18 20:10:24.678990725 (195h59m27.694660326s) [noise=true , address=77.90.31.14:4662 , piece_id=LMB5VH2XB2ZQ5DEO3H5CHWSX4R2GDP4FNQFSFWY32LJ5KEYBRJPQ, seg_pos={0 0}, upload=188260]: Done uploading piece. err: <nil>
Mar 18 20:10:24.684137779 (195h59m27.699807368s) [noise=true , address=45.91.116.5:4301 , piece_id=HSBVXCAHRZRFTYYNLX7DBJ6FIJJLFRNSMJXFSHFBSXB7OQEBRCPA, seg_pos={0 0}, upload=188260]: Done uploading piece. err: <nil>
Mar 18 20:10:24.703405821 (195h59m27.719075422s) [noise=true , address=45.91.116.13:4157 , piece_id=UD6SW24UZINEIDS5IQBENER7Y4YEAZHZ26CTPF6TJ24UCB7XGF6Q, seg_pos={0 0}, upload=188260]: Done uploading piece. err: <nil>
Mar 18 20:10:24.727977058 (195h59m27.743646659s) [noise=true , address=79.127.219.45:4855 , piece_id=CU4IERVVHAUG6XXXUP6MHYSSDQMQX2Z3KWPP6QZSB6PCGHRC4UIQ, seg_pos={0 0}, upload=188260]: Done uploading piece. err: <nil>
Mar 18 20:10:50.634951010 (195h59m53.650620623s) [noise=true , address=145.79.227.7:4295 , piece_id=DMCF33ZZFY4RXZMYKZWPC3DS2VZXCMFENRLPA6PAJHVL3W6Y2BSA, seg_pos={0 0}, upload=188260]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:10:50.635037857 (195h59m53.650707463s) [noise=true , address=145.79.227.43:4158 , piece_id=7WBRLGW753FYRJJTT4LPNGG622YBQUZDDTQZY37FRSY5ISK3UJSA, seg_pos={0 0}, upload=188260]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: rpc: dial tcp 145.79.227.43:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:10:50.635165813 (195h59m53.650835416s) [noise=true , address=145.79.227.36:4521 , piece_id=MESIVF7I3PLOSK47LO4VMTMFPHN737AFCA5BQIQQRZPTS3YSK23A, seg_pos={0 0}, upload=188260]: Done uploading piece. err: ecclient: upload failed (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD, address:145.79.227.36:4521): protocol: rpc: dial tcp 145.79.227.36:4521: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:10:50.635388032 (195h59m53.651057637s) [noise=true , address=145.79.227.9:4368 , piece_id=JXSP3N56X2UDFQWGCXNPDMFKKJLK7HFU336Q7KUTE3H2R3I5TR5A, seg_pos={0 0}, upload=188260]: Done uploading piece. err: ecclient: upload failed (node:12ByhoW6Effjs8acfsvdF1xzDwBVqggsBK53oRZ8bYhEYYPChKz, address:145.79.227.9:4368): protocol: rpc: dial tcp 145.79.227.9:4368: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:10:50.635516887 (195h59m53.651186492s) [noise=true , address=145.79.227.21:4473 , piece_id=3EKSQJEBR2KC4OPW2SPTJPCD6CS7IMDRXW4TZ63XQPLPBWTEEDRA, seg_pos={0 0}, upload=188260]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: rpc: dial tcp 145.79.227.21:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:10:50.635624295 (195h59m53.651293898s) [noise=true , address=145.79.227.42:4122 , piece_id=TR3IK2MPYDZXWII4PTFBBY2QNWBXQPOP4A7KRJS7CYK2V6EF3OTA, seg_pos={0 0}, upload=188260]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:10:50.635649711 (195h59m53.651319301s) [seg_pos={0 0}, upload=188260]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Mar 18 20:10:50.745607337 (195h59m53.761276929s) [upload=188260]: Done uploading segments...
Mar 18 20:10:50.745654115 (195h59m53.761323703s) [upload=188260]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.112268196s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [................XXX.XXX...............X.....................] duration:
13.883769503s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:10:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.883872221s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [..........XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001395702s trace:
https://jaeger.dev.storj.io/trace/257d2e78e25454e3
FAIL (2026-03-18 20:10:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001418206s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
45 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX] duration:
30.113057092s trace:
https://jaeger.dev.storj.io/trace/1bfb09a092211b42
FAIL (2026-03-18 20:10:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Mar 18 20:09:50.645767842 (195h58m53.661437448s) [upload=188250]: Starting upload
Mar 18 20:09:50.645819278 (195h58m53.661488883s) [upload=188250]: Done starting upload
Mar 18 20:09:50.646111146 (195h58m53.661780733s) [upload=188250]: Uploading segments...
Mar 18 20:09:50.646183794 (195h58m53.661853393s) [upload=188250]: Got next segment. Inline: false
Mar 18 20:09:50.646216259 (195h58m53.661885846s) [upload=188250]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:09:50.646220946 (195h58m53.661890545s) [upload=188250]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:09:54.394817321 (195h58m57.410486921s) [seg_pos={0 0}, upload=188250]: Begin upload segment...
Mar 18 20:09:54.394882671 (195h58m57.410552261s) [seg_pos={0 0}, upload=188250]: Done begin upload segment.
Mar 18 20:09:54.394911736 (195h58m57.410581339s) [noise=true , address=77.90.31.17:4774 , piece_id=6FVDJF7PKBSAE6QDSEJEK4H3L7X5G7MUBETYI4C2I36JSLOZUPUQ, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.396108328 (195h58m57.41177794s) [noise=true , address=45.91.116.10:4083 , piece_id=WCMJYAETN3XRXABQOZOXBNCVAIZDBKWYXHMINJU4OEPCKQXQ74ZA, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.396323402 (195h58m57.411993013s) [noise=true , address=145.79.227.25:4620 , piece_id=LQKNIB2BDZJYYUOSRCXO554OZC5RV742LM5JKDHAVI2LDB2ZJSTA, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.396645049 (195h58m57.412314659s) [noise=true , address=145.79.227.5:4221 , piece_id=6Q6A4X22YRKM3JW7ZIZLKUHNSYUZH3KFJRA3TMMG43P6NGL2HDSQ, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.396846231 (195h58m57.412515836s) [noise=true , address=45.91.116.14:4193 , piece_id=TJRT7ZPYTBY7IJ7XZN6EOH7WDXDDTFOPNFGJE4ZEOUGBBKRHJRUA, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.397053126 (195h58m57.41272274s) [noise=true , address=145.79.227.35:4482 , piece_id=SSBGRB4LFJMBMF5HGQUR4ZKQXCNUJS2BBSG2WN5XLPU4OWYCXI7Q, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.397368703 (195h58m57.413038315s) [noise=true , address=79.127.219.45:4855 , piece_id=BIPPF4ZDTF7EJZASIXGRHF3V7LR7SEYY4ZHDWV7DQCRKDWZVVU4A, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.397462852 (195h58m57.41313246s) [noise=true , address=145.79.227.33:4871 , piece_id=3WEZCRILXN4MLPACRUDRZ66XS2ZPMHJ2OK2XY63YV3O2EYM3SHEA, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.398119935 (195h58m57.413789565s) [noise=true , address=45.91.116.7:4009 , piece_id=WHGJ5IXCLRNRMYC4OZRNKHQAFYIQ2CAFL6GH5XFA4GY35UYIUYUQ, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.398155595 (195h58m57.413825207s) [noise=true , address=45.91.116.4:4952 , piece_id=F5TD7XX45OTJ6HSBWONX2IVD5Y64P5UW45PK47NICM3GJSNQFVJA, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.398692414 (195h58m57.414362025s) [noise=true , address=79.127.219.46:4891 , piece_id=RJGCOW4MV366NR4I2LEYJYDYGUENQVEDTSULABUTJ3IIP3AGJ64A, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.399280537 (195h58m57.414950148s) [noise=true , address=45.91.116.13:4155 , piece_id=OOKIQAB4KY2OA6SRAPPZOAHP4E3O7Y6K3PFQGFJ6AKZKN5KH5AVA, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.399914642 (195h58m57.415584349s) [noise=true , address=77.90.31.16:4738 , piece_id=W5LMLSNXA4RGBA5OVUQTMOP6STJRRRVFBMFLILU72L2YTQ2EQIAA, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.401055494 (195h58m57.416725096s) [noise=true , address=145.79.227.9:4371 , piece_id=DRAJL742XJW7PRBEXX6Y64NFHWEKFPNLIXAQ5T7LPGTTP3J65O4Q, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.402857480 (195h58m57.418527089s) [noise=true , address=77.90.31.12:4590 , piece_id=SLH3LDO7ASRBQRBIS4DLPXZ6CU7COVRCZE5EMU36BOVK4PPVL25A, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.403570371 (195h58m57.419239983s) [noise=true , address=145.79.227.31:4800 , piece_id=VTXPN6URLIMS6D24SFBH3TATHFIQ6L4GCMW4VPRF5ITTXA646KLQ, seg_pos={0 0}, upload=188250]: Uploading piece...
Mar 18 20:09:54.988735940 (195h58m58.004405551s) [upload=188250]: Next returned nil segment
Mar 18 20:09:54.988755702 (195h58m58.004425308s) [upload=188250]: Waiting for error group managing segments...
Mar 18 20:09:55.134739677 (195h58m58.150409284s) [noise=true , address=45.91.116.7:4009 , piece_id=WHGJ5IXCLRNRMYC4OZRNKHQAFYIQ2CAFL6GH5XFA4GY35UYIUYUQ, seg_pos={0 0}, upload=188250]: Done uploading piece. err: <nil>
Mar 18 20:09:55.137747173 (195h58m58.153416774s) [noise=true , address=77.90.31.17:4774 , piece_id=6FVDJF7PKBSAE6QDSEJEK4H3L7X5G7MUBETYI4C2I36JSLOZUPUQ, seg_pos={0 0}, upload=188250]: Done uploading piece. err: <nil>
Mar 18 20:09:55.138543229 (195h58m58.154212843s) [noise=true , address=45.91.116.10:4083 , piece_id=WCMJYAETN3XRXABQOZOXBNCVAIZDBKWYXHMINJU4OEPCKQXQ74ZA, seg_pos={0 0}, upload=188250]: Done uploading piece. err: <nil>
Mar 18 20:09:55.141983675 (195h58m58.157653275s) [noise=true , address=77.90.31.16:4738 , piece_id=W5LMLSNXA4RGBA5OVUQTMOP6STJRRRVFBMFLILU72L2YTQ2EQIAA, seg_pos={0 0}, upload=188250]: Done uploading piece. err: <nil>
Mar 18 20:09:55.145254955 (195h58m58.160924563s) [noise=true , address=79.127.219.46:4891 , piece_id=RJGCOW4MV366NR4I2LEYJYDYGUENQVEDTSULABUTJ3IIP3AGJ64A, seg_pos={0 0}, upload=188250]: Done uploading piece. err: <nil>
Mar 18 20:09:55.155750590 (195h58m58.1714202s) [noise=true , address=45.91.116.14:4193 , piece_id=TJRT7ZPYTBY7IJ7XZN6EOH7WDXDDTFOPNFGJE4ZEOUGBBKRHJRUA, seg_pos={0 0}, upload=188250]: Done uploading piece. err: <nil>
Mar 18 20:09:55.160231618 (195h58m58.175901219s) [noise=true , address=45.91.116.4:4952 , piece_id=F5TD7XX45OTJ6HSBWONX2IVD5Y64P5UW45PK47NICM3GJSNQFVJA, seg_pos={0 0}, upload=188250]: Done uploading piece. err: <nil>
Mar 18 20:09:55.160963066 (195h58m58.176632667s) [noise=true , address=45.91.116.13:4155 , piece_id=OOKIQAB4KY2OA6SRAPPZOAHP4E3O7Y6K3PFQGFJ6AKZKN5KH5AVA, seg_pos={0 0}, upload=188250]: Done uploading piece. err: <nil>
Mar 18 20:09:55.163288197 (195h58m58.178957797s) [noise=true , address=77.90.31.12:4590 , piece_id=SLH3LDO7ASRBQRBIS4DLPXZ6CU7COVRCZE5EMU36BOVK4PPVL25A, seg_pos={0 0}, upload=188250]: Done uploading piece. err: <nil>
Mar 18 20:09:55.203718157 (195h58m58.219387763s) [noise=true , address=79.127.219.45:4855 , piece_id=BIPPF4ZDTF7EJZASIXGRHF3V7LR7SEYY4ZHDWV7DQCRKDWZVVU4A, seg_pos={0 0}, upload=188250]: Done uploading piece. err: <nil>
Mar 18 20:10:20.356352814 (195h59m23.372022424s) [noise=true , address=145.79.227.35:4482 , piece_id=SSBGRB4LFJMBMF5HGQUR4ZKQXCNUJS2BBSG2WN5XLPU4OWYCXI7Q, seg_pos={0 0}, upload=188250]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: rpc: dial tcp 145.79.227.35:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:10:20.356618597 (195h59m23.372288208s) [noise=true , address=145.79.227.5:4221 , piece_id=6Q6A4X22YRKM3JW7ZIZLKUHNSYUZH3KFJRA3TMMG43P6NGL2HDSQ, seg_pos={0 0}, upload=188250]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:10:20.356912382 (195h59m23.372581993s) [noise=true , address=145.79.227.9:4371 , piece_id=DRAJL742XJW7PRBEXX6Y64NFHWEKFPNLIXAQ5T7LPGTTP3J65O4Q, seg_pos={0 0}, upload=188250]: Done uploading piece. err: ecclient: upload failed (node:12cSqcGZqvZqs1QKBi6WK4fUDKgvQcmTupR1C9ZkWz879XeWPEL, address:145.79.227.9:4371): protocol: rpc: dial tcp 145.79.227.9:4371: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:10:20.357194742 (195h59m23.372864354s) [noise=true , address=145.79.227.25:4620 , piece_id=LQKNIB2BDZJYYUOSRCXO554OZC5RV742LM5JKDHAVI2LDB2ZJSTA, seg_pos={0 0}, upload=188250]: Done uploading piece. err: ecclient: upload failed (node:12QahNthBhqWBPgFFwAbsy5VrEuMxBkcuybmabY43sqBKKrW6ev, address:145.79.227.25:4620): protocol: rpc: dial tcp 145.79.227.25:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:10:20.357435004 (195h59m23.373104615s) [noise=true , address=145.79.227.31:4800 , piece_id=VTXPN6URLIMS6D24SFBH3TATHFIQ6L4GCMW4VPRF5ITTXA646KLQ, seg_pos={0 0}, upload=188250]: Done uploading piece. err: ecclient: upload failed (node:121yHupysREmASSVx3WUN57duu16QEs7sA2BPBWyaThFzruvch2, address:145.79.227.31:4800): protocol: rpc: dial tcp 145.79.227.31:4800: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:10:20.357650213 (195h59m23.373319801s) [noise=true , address=145.79.227.33:4871 , piece_id=3WEZCRILXN4MLPACRUDRZ66XS2ZPMHJ2OK2XY63YV3O2EYM3SHEA, seg_pos={0 0}, upload=188250]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:145.79.227.33:4871): protocol: rpc: dial tcp 145.79.227.33:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:10:20.357696344 (195h59m23.373365949s) [seg_pos={0 0}, upload=188250]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Mar 18 20:10:20.464336649 (195h59m23.48000626s) [upload=188250]: Done uploading segments...
Mar 18 20:10:20.464457089 (195h59m23.480126698s) [upload=188250]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.113086252s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [...............X..X...X..X..X........X..X..X...X............] duration:
3.741287751s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:10:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.741311431s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.................XXX.XXX...............X....................] duration:
18.95698628s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:10:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.957013063s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [................X..X...X..X..X........X..X..X...X...........] duration:
18.367166673s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:10:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 18.367192438s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [...........XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001446093s trace:
https://jaeger.dev.storj.io/trace/1a1ea872052b1730
FAIL (2026-03-18 20:09:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001475584s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [..................XXX.XXX...............X...................] duration:
17.386741861s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:09:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.38678746s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
44 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXX] duration:
30.113365448s trace:
https://jaeger.dev.storj.io/trace/5b26b9a3b08974aa
FAIL (2026-03-18 20:09:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Mar 18 20:09:14.217057253 (195h58m17.232726853s) [upload=188239]: Starting upload
Mar 18 20:09:14.217130716 (195h58m17.232800322s) [upload=188239]: Done starting upload
Mar 18 20:09:14.217329112 (195h58m17.232998702s) [upload=188239]: Uploading segments...
Mar 18 20:09:14.217435859 (195h58m17.233105459s) [upload=188239]: Got next segment. Inline: false
Mar 18 20:09:14.217489196 (195h58m17.233158784s) [upload=188239]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:09:14.217494423 (195h58m17.233164024s) [upload=188239]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:09:17.353189346 (195h58m20.368858952s) [seg_pos={0 0}, upload=188239]: Begin upload segment...
Mar 18 20:09:17.353257321 (195h58m20.368926922s) [seg_pos={0 0}, upload=188239]: Done begin upload segment.
Mar 18 20:09:17.353346296 (195h58m20.369015899s) [noise=true , address=45.91.116.5:4301 , piece_id=7YHZYPAO5GIFGQ26GOATPOYUUQTBS7US2BVCSSMEDUMLKCIBNOOA, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.353362363 (195h58m20.369031964s) [noise=true , address=145.79.227.25:4620 , piece_id=4LXPBGWVWKYX4BUBPDJY4W4JVODFMXYKETHDTV5PNRH4S7UVNI4A, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.354300657 (195h58m20.369970267s) [noise=true , address=79.127.219.41:4925 , piece_id=5BFETNZ626OB5BNW66YXNTK47JRE5N62BVIX5QDVRG445W7Z7JRQ, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.354816748 (195h58m20.370486359s) [noise=true , address=77.90.31.18:4805 , piece_id=5TZZS2FWN7YRK7MDD525IMKRONGOGEO3CTYXR4MOXMD6YT5R4TCA, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.355229086 (195h58m20.370898686s) [noise=true , address=45.91.116.13:4155 , piece_id=EF7GOQYA7M53LVPLOVUT6APZ2VMKHX72XY7LRTNNDE3GYL7UQHMQ, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.357959057 (195h58m20.373628667s) [noise=true , address=45.91.116.4:4955 , piece_id=4HYRKJTV7CXAXR4BHKLID5P6I2CJV2EU4JRQRZ6KEF5O6EWAC6KA, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.358807730 (195h58m20.374477338s) [noise=true , address=77.90.31.15:4702 , piece_id=TUBJMZU6MPTNCUWAP46AN4S4SF65LTESKWXMGOCXFXZU2CEATAEA, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.361012521 (195h58m20.376682132s) [noise=true , address=145.79.227.48:4306 , piece_id=BYK6BZRPJL3CBKVQHFQJJRAS7G5N4V6TE24R6ME6QYWNS7EORSHQ, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.366439887 (195h58m20.382109497s) [noise=true , address=45.91.116.11:4152 , piece_id=UQRDW6LKQIM33K63KWAO7QVNSDBX7Q5LAXOEZKSFNU4TE5WQZ7YQ, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.370336906 (195h58m20.386006521s) [noise=true , address=45.91.116.14:4193 , piece_id=EHXDEY5VXSAGD4LU74THGBV6LQHLDKXJM4GHTGYP6LM72J75PV3Q, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.373720969 (195h58m20.389390581s) [noise=true , address=45.91.116.8:4045 , piece_id=OJ2LM4RTDHZAIKSBZSQJVKNUSADUGR4BC6I2CGIAKCGXW2FQFXMQ, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.374810481 (195h58m20.390480093s) [noise=true , address=77.90.31.19:4845 , piece_id=DLHEYLOV4AUZHDT3ZM4SF3T5BLL3URRXKYRSKPNSBW76IQFOZXJQ, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.375780250 (195h58m20.39144986s) [noise=true , address=145.79.227.20:4438 , piece_id=6O5VIXJ4KBLSZH7BZP2G4ZG4BRHJJI7ZVGJZBID767ZYH4T6WWTA, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.376828152 (195h58m20.392497763s) [noise=true , address=145.79.227.42:4121 , piece_id=2JY4SNXKSEQLPO7AOLVJADY3EQ6IELSDC6AEUSWRMCV5QQTJQXBQ, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.380317231 (195h58m20.395986834s) [noise=true , address=77.90.31.16:4735 , piece_id=ONAMLMMB7Q63NVOKFXXWAAOAQ62V54ZWKPDENEUUCSIR25L6GRRQ, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.383429623 (195h58m20.399099233s) [noise=true , address=145.79.227.36:4522 , piece_id=N5DKCHDSM65THH5VMQCDAOCIMGPMD73GUK7AVJCYAUVJ3Q2LAT2Q, seg_pos={0 0}, upload=188239]: Uploading piece...
Mar 18 20:09:17.966530047 (195h58m20.982199719s) [upload=188239]: Next returned nil segment
Mar 18 20:09:17.966554693 (195h58m20.982224324s) [upload=188239]: Waiting for error group managing segments...
Mar 18 20:09:18.084999716 (195h58m21.100669321s) [noise=true , address=77.90.31.18:4805 , piece_id=5TZZS2FWN7YRK7MDD525IMKRONGOGEO3CTYXR4MOXMD6YT5R4TCA, seg_pos={0 0}, upload=188239]: Done uploading piece. err: <nil>
Mar 18 20:09:18.098940415 (195h58m21.11461002s) [noise=true , address=79.127.219.41:4925 , piece_id=5BFETNZ626OB5BNW66YXNTK47JRE5N62BVIX5QDVRG445W7Z7JRQ, seg_pos={0 0}, upload=188239]: Done uploading piece. err: <nil>
Mar 18 20:09:18.101374094 (195h58m21.117043694s) [noise=true , address=77.90.31.16:4735 , piece_id=ONAMLMMB7Q63NVOKFXXWAAOAQ62V54ZWKPDENEUUCSIR25L6GRRQ, seg_pos={0 0}, upload=188239]: Done uploading piece. err: <nil>
Mar 18 20:09:18.113081907 (195h58m21.128751511s) [noise=true , address=45.91.116.13:4155 , piece_id=EF7GOQYA7M53LVPLOVUT6APZ2VMKHX72XY7LRTNNDE3GYL7UQHMQ, seg_pos={0 0}, upload=188239]: Done uploading piece. err: <nil>
Mar 18 20:09:18.113369874 (195h58m21.129039485s) [noise=true , address=45.91.116.11:4152 , piece_id=UQRDW6LKQIM33K63KWAO7QVNSDBX7Q5LAXOEZKSFNU4TE5WQZ7YQ, seg_pos={0 0}, upload=188239]: Done uploading piece. err: <nil>
Mar 18 20:09:18.116985040 (195h58m21.132654646s) [noise=true , address=45.91.116.14:4193 , piece_id=EHXDEY5VXSAGD4LU74THGBV6LQHLDKXJM4GHTGYP6LM72J75PV3Q, seg_pos={0 0}, upload=188239]: Done uploading piece. err: <nil>
Mar 18 20:09:18.118432720 (195h58m21.134102336s) [noise=true , address=45.91.116.4:4955 , piece_id=4HYRKJTV7CXAXR4BHKLID5P6I2CJV2EU4JRQRZ6KEF5O6EWAC6KA, seg_pos={0 0}, upload=188239]: Done uploading piece. err: <nil>
Mar 18 20:09:18.125469705 (195h58m21.141139305s) [noise=true , address=77.90.31.15:4702 , piece_id=TUBJMZU6MPTNCUWAP46AN4S4SF65LTESKWXMGOCXFXZU2CEATAEA, seg_pos={0 0}, upload=188239]: Done uploading piece. err: <nil>
Mar 18 20:09:18.144756373 (195h58m21.160425976s) [noise=true , address=45.91.116.5:4301 , piece_id=7YHZYPAO5GIFGQ26GOATPOYUUQTBS7US2BVCSSMEDUMLKCIBNOOA, seg_pos={0 0}, upload=188239]: Done uploading piece. err: <nil>
Mar 18 20:09:18.144859059 (195h58m21.160528649s) [noise=true , address=45.91.116.8:4045 , piece_id=OJ2LM4RTDHZAIKSBZSQJVKNUSADUGR4BC6I2CGIAKCGXW2FQFXMQ, seg_pos={0 0}, upload=188239]: Done uploading piece. err: <nil>
Mar 18 20:09:26.378359510 (195h58m29.394029111s) [noise=true , address=77.90.31.19:4845 , piece_id=DLHEYLOV4AUZHDT3ZM4SF3T5BLL3URRXKYRSKPNSBW76IQFOZXJQ, seg_pos={0 0}, upload=188239]: Done uploading piece. err: <nil>
Mar 18 20:09:43.914584773 (195h58m46.930254379s) [noise=true , address=145.79.227.25:4620 , piece_id=4LXPBGWVWKYX4BUBPDJY4W4JVODFMXYKETHDTV5PNRH4S7UVNI4A, seg_pos={0 0}, upload=188239]: Done uploading piece. err: ecclient: upload failed (node:12QahNthBhqWBPgFFwAbsy5VrEuMxBkcuybmabY43sqBKKrW6ev, address:145.79.227.25:4620): protocol: rpc: dial tcp 145.79.227.25:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:09:43.914603323 (195h58m46.930272932s) [noise=true , address=145.79.227.36:4522 , piece_id=N5DKCHDSM65THH5VMQCDAOCIMGPMD73GUK7AVJCYAUVJ3Q2LAT2Q, seg_pos={0 0}, upload=188239]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:145.79.227.36:4522): protocol: rpc: dial tcp 145.79.227.36:4522: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:09:43.914765121 (195h58m46.930434731s) [noise=true , address=145.79.227.42:4121 , piece_id=2JY4SNXKSEQLPO7AOLVJADY3EQ6IELSDC6AEUSWRMCV5QQTJQXBQ, seg_pos={0 0}, upload=188239]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:09:43.914908627 (195h58m46.930578236s) [noise=true , address=145.79.227.48:4306 , piece_id=BYK6BZRPJL3CBKVQHFQJJRAS7G5N4V6TE24R6ME6QYWNS7EORSHQ, seg_pos={0 0}, upload=188239]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: rpc: dial tcp 145.79.227.48:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:09:43.915208643 (195h58m46.930878246s) [noise=true , address=145.79.227.20:4438 , piece_id=6O5VIXJ4KBLSZH7BZP2G4ZG4BRHJJI7ZVGJZBID767ZYH4T6WWTA, seg_pos={0 0}, upload=188239]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: rpc: dial tcp 145.79.227.20:4438: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Mar 18 20:09:43.915241620 (195h58m46.930911223s) [seg_pos={0 0}, upload=188239]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Mar 18 20:09:44.024663976 (195h58m47.040333582s) [upload=188239]: Done uploading segments...
Mar 18 20:09:44.024822551 (195h58m47.040492157s) [upload=188239]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.113425102s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [............XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000954475s trace:
https://jaeger.dev.storj.io/trace/69c554f6b4e00eb9
FAIL (2026-03-18 20:09:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000976246s
core
uplink
us-select-1
us1-placement-20
|