Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X....................X..X..X.............X......X.X.....X...] duration: 7.609092175s FAIL (2025-07-09 02:01:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.609119505s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 24.63942333s FAIL (2025-07-09 02:00:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 24.639477011s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [.......X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.] duration: 48.555411136s FAIL (2025-07-09 02:00:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.555439774s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX....................X..X..X.............X......X.X.....X..] duration: 9.702021648s FAIL (2025-07-09 02:00:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.702056819s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX] duration: 1m0.001059622s FAIL (2025-07-09 01:59:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001081668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...................................X.......................] duration: 2.290892336s FAIL (2025-07-09 01:59:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.290923337s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX....................X..X..X.............X......X.X.....X.] duration: 12.232556759s FAIL (2025-07-09 01:58:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.232583732s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...................................X......................] duration: 2.459360949s FAIL (2025-07-09 01:58:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.459395402s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XX....................X..X..X.............X......X.X.....X] duration: 30.000773129s FAIL (2025-07-09 01:58:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000814637s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [.........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XX] duration: 1m0.001109915s FAIL (2025-07-09 01:58:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001122435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...................................X.....................] duration: 2.300844908s FAIL (2025-07-09 01:58:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.300876861s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [X.........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.X] duration: 1m0.000760191s FAIL (2025-07-09 01:57:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000777565s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...................................X....................] duration: 2.373028223s FAIL (2025-07-09 01:56:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.373063579s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...XX....................X..X..X.............X......X.X.....] duration: 11.200215344s FAIL (2025-07-09 01:56:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.200240883s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...................................X...................] duration: 26.338431942s FAIL (2025-07-09 01:56:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 26.338455068s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [.X.........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.] duration: 52.726498944s FAIL (2025-07-09 01:56:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.726570873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....XX....................X..X..X.............X......X.X....] duration: 3.339559362s FAIL (2025-07-09 01:55:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.339586753s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...................................X..................] duration: 21.116736008s FAIL (2025-07-09 01:55:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 21.116838629s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [..X.........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX] duration: 1m0.000809129s FAIL (2025-07-09 01:55:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000827997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....XX....................X..X..X.............X......X.X...] duration: 13.012502194s FAIL (2025-07-09 01:55:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.012533091s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...................................X.................] duration: 29.376695225s FAIL (2025-07-09 01:54:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 29.37673278s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [...X.........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XX] duration: 1m0.001910917s FAIL (2025-07-09 01:54:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001930902s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......XX....................X..X..X.............X......X.X..] duration: 2.707670339s FAIL (2025-07-09 01:53:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.70772617s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...................................X................] duration: 2.340834154s FAIL (2025-07-09 01:53:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.3408686s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [X...X.........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..X] duration: 1m0.001602864s FAIL (2025-07-09 01:53:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001623499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......XX....................X..X..X.............X......X.X.] duration: 20.945773845s FAIL (2025-07-09 01:53:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.94580953s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...................................X...............] duration: 2.273950359s FAIL (2025-07-09 01:52:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.27398123s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...................................X..............] duration: 2.379609449s FAIL (2025-07-09 01:52:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.37964247s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [XX...X.........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..] duration: 48.124721065s FAIL (2025-07-09 01:52:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.12474554s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.......XX....................X..X..X.............X......X.X] duration: 30.001347098s FAIL (2025-07-09 01:52:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001381419s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...................................X.............] duration: 2.421575962s FAIL (2025-07-09 01:51:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.421607108s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [XXX...X.........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX.] duration: 56.923023708s FAIL (2025-07-09 01:51:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.923040002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX.......XX....................X..X..X.............X......X.] duration: 23.474725167s FAIL (2025-07-09 01:51:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.474793419s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXX.......XX....................X..X..X.............X......X] duration: 30.001528835s FAIL (2025-07-09 01:50:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00155443s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...................................X............] duration: 2.299708514s FAIL (2025-07-09 01:50:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.29974603s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [XXXX...X.........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX] duration: 1m0.001890005s FAIL (2025-07-09 01:50:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001911781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...................................X...........] duration: 2.503185014s FAIL (2025-07-09 01:50:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.503220055s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXXX.......XX....................X..X..X.............X......] duration: 23.110715083s FAIL (2025-07-09 01:50:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.110748095s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...................................X..........] duration: 2.477796124s FAIL (2025-07-09 01:49:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.477824213s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [XXXXX...X.........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXX] duration: 1m0.000882618s FAIL (2025-07-09 01:49:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000910957s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXXX.......XX....................X..X..X.............X.....] duration: 20.231060263s FAIL (2025-07-09 01:49:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.23113826s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...................................X.........] duration: 2.347148177s FAIL (2025-07-09 01:48:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.347188693s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [.XXXXX...X.........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XX] duration: 1m0.00096321s FAIL (2025-07-09 01:48:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000978558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XXXXX.......XX....................X..X..X.............X....] duration: 21.544182182s FAIL (2025-07-09 01:47:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.544448255s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.XXXXX.......XX....................X..X..X.............X...] duration: 17.100064502s FAIL (2025-07-09 01:47:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.100089881s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...................................X........] duration: 2.528876096s FAIL (2025-07-09 01:47:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.52890967s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [..XXXXX...X.........X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.X] duration: 1m0.001240013s FAIL (2025-07-09 01:47:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001280625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.XXXXX.......XX....................X..X..X.............X..] duration: 2.888267111s FAIL (2025-07-09 01:46:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.888289568s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...................................X.......] duration: 2.251857773s FAIL (2025-07-09 01:46:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.251909844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...................................X......] duration: 2.31249852s FAIL (2025-07-09 01:46:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments... Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil> Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments... Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment... Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment. Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece... Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments... Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil> Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments... Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.312531759s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)