Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [X...............X...........................................] duration:
3.336052098s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:26:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.336159136s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X.............] duration:
2.266855326s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:26:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.267016851s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX............] duration:
2.690934011s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:26:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.690956306s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [.X..X..X........X..X..X...X.................................] duration:
3.330278399s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:26:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.330380735s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX...........] duration:
2.3784398s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:26:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.37847553s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X............] duration:
1.706049367s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:25:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.706113471s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [XX...............X..........................................] duration:
2.882133946s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:25:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.88217417s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX..........] duration:
2.332824478s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:25:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.33285512s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [..X..X..X........X..X..X...X................................] duration:
3.162103636s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:25:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.162134517s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [XXX...............X.........................................] duration:
2.878664866s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:24:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.878700989s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X...........] duration:
2.50008584s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:24:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.500110803s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX.........] duration:
2.334711517s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:24:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.334758976s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..........] duration:
1.635349234s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:24:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.63538213s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [...X..X..X........X..X..X...X...............................] duration:
3.160279917s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:24:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.160318101s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.XXX...............X........................................] duration:
3.447480588s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:24:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.44751177s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X.........] duration:
2.173669146s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:24:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.1736994s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX........] duration:
2.401805309s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:23:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.401836678s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X........] duration:
2.113808888s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:23:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.113839247s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [X...X..X..X........X..X..X...X..............................] duration:
3.042913133s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:23:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.042939308s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [X.XXX...............X.......................................] duration:
2.853475865s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:23:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.853506436s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [.X...X..X..X........X..X..X...X.............................] duration:
2.99313045s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:23:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.993163453s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [XX.XXX...............X......................................] duration:
2.947743375s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:23:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.947768455s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [..XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X.......] duration:
1.632571183s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:23:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.632602724s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX.......] duration:
2.526180501s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:22:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.526204072s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [XXX.XXX...............X.....................................] duration:
2.786467568s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:22:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.786547131s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [...XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X......] duration:
1.868743262s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:22:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.868820396s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [..X...X..X..X........X..X..X...X............................] duration:
3.171768353s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:22:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.171799146s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X.....] duration:
1.851535912s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:22:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.851592058s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [X....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X....] duration:
2.276083393s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:21:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.276101147s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.XXX.XXX...............X....................................] duration:
2.926869653s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:21:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.92706995s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX......] duration:
2.416075307s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:21:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.416108866s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [X..X...X..X..X........X..X..X...X...........................] duration:
3.293668262s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:21:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.293705252s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [XX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X...] duration:
1.605326869s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:21:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.605843464s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [..XXX.XXX...............X...................................] duration:
2.773753414s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:21:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.774229313s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX.....] duration:
2.407074864s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:21:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.407109885s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.X..X...X..X..X........X..X..X...X..........................] duration:
3.177860761s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:20:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.177882356s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X..] duration:
1.580078835s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:20:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.580368729s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [...XXX.XXX...............X..................................] duration:
2.907845992s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:20:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.907887003s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X.] duration:
2.767606567s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:20:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.767643908s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [..X..X...X..X..X........X..X..X...X.........................] duration:
3.159298669s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:20:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.15933225s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX....] duration:
2.356873696s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:19:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.356902595s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [....XXX.XXX...............X.................................] duration:
2.911133648s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:19:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.911229499s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX...] duration:
2.577513247s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:19:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.577551585s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [...X..X...X..X..X........X..X..X...X........................] duration:
3.189545821s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:19:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.189572344s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....X] duration:
5.00125825s trace:
https://jaeger.dev.storj.io/trace/6f0bd8dc5877663a
FAIL (2026-03-18 20:19:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001283739s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [....X..X...X..X..X........X..X..X...X.......................] duration:
3.170790991s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:19:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.170821126s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.....XXX.XXX...............X................................] duration:
2.875109992s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:19:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.875135351s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX....] duration:
2.358469409s trace:
https://jaeger.dev.storj.io/trace/778e2ecfec472f01
FAIL (2026-03-18 20:19:06 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.358507621s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XXX..] duration:
2.495999299s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:18:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.49602664s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.....X..X...X..X..X........X..X..X...X......................] duration:
3.118843255s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:18:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.118909364s
core
uplink
us-select-1
us1-placement-20
|