Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 40 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X.XX.............] duration: 884.210626ms FAIL (2025-08-07 21:55:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 884.262707ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX........] duration: 2.433070663s FAIL (2025-08-07 21:55:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.433104299s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.......................XX.X..........X.....X.........X] duration: 2.218580245s FAIL (2025-08-07 21:55:08 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== stItem_ObjectBegin Aug 7 21:55:06.961424499 (12h15m28.70548136s) [upload=12632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:55:07.163785978 (12h15m28.90784284s) [seg_pos={0 0}, upload=12632]: Begin upload segment... Aug 7 21:55:07.163841484 (12h15m28.907898333s) [seg_pos={0 0}, upload=12632]: Done begin upload segment. Aug 7 21:55:07.163912678 (12h15m28.907969541s) [noise=true , address=104.36.82.45:4937 , piece_id=CEXRQRHANULB5PPL6ZLYHGBEAOLSEMS4VXNEM4JB4ARQY5CUNYAA, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.163925769 (12h15m28.907982631s) [noise=true , address=104.36.82.23:4478 , piece_id=C56Z7JYNDZLYTNLKAVBM4FUWPFGEX6DOZVPEISDKOWE64VRARUVA, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.164335012 (12h15m28.908391883s) [noise=true , address=104.36.82.11:4143 , piece_id=BYKKXKVMC2VEAZFQF5MOZDQGDWHJ4HQUZTR34QJMY4H4AJ42D33Q, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.164397043 (12h15m28.908453906s) [noise=true , address=104.36.82.56:4233 , piece_id=33U5NWSQBMNN4GPUPNSVYFP7CXPV4BXMZMNOAAOMN6J3U62BCCBA, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.164681548 (12h15m28.908738412s) [noise=true , address=104.36.82.4:4825 , piece_id=FYBWSKJSJ47XJ7YEHK3ZW5LGXNS6TWE4EWE2QUEPEAKYXAYWYOTQ, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.164816506 (12h15m28.908873369s) [noise=true , address=79.127.205.233:4634 , piece_id=MPFJGGY2DQEUOJTI6JHAPEMWVYBU3ULOFAPHCFBDKV7KUJVZ3S2A, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.164988405 (12h15m28.909045257s) [noise=true , address=104.36.82.59:4324 , piece_id=HGM5FKLCSIZ3RTD3TFSADP2CMBRCWHZ332Y2VMJEYF2THB74MFVA, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.165162141 (12h15m28.909219004s) [noise=true , address=104.36.82.61:4407 , piece_id=KKBMJS2DYMHPJJKF7PCJX763TT3OVTOIMA3XX2K6UOX5KW2B4E5Q, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.165569008 (12h15m28.90962587s) [noise=true , address=104.36.82.5:4872 , piece_id=ROROCMYX65Y2FJOOQZTHU3VI4KHVGEH3AONGULBWRMBVBAH6MWVQ, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.170705288 (12h15m28.914762156s) [noise=true , address=104.36.82.60:4354 , piece_id=AE4HFMEQGKPMDR7FVC3JNM5YIZMQ6W2S5ZXBYCME2CEC7V6J5LZQ, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.191355319 (12h15m28.935412189s) [noise=true , address=104.36.82.9:4012 , piece_id=HSE3U5ET7TC5SLU6CENH5IUZ2SCAE2ZNYCXQHXV4VKY6LN6WTNIQ, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.193899880 (12h15m28.937956749s) [noise=true , address=104.36.82.37:4679 , piece_id=YRR2CJYFY6DJVE7JCAFC5OUQWODTR7UQXVGG6FXBPPUQW2TVCFHQ, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.194999854 (12h15m28.939056717s) [noise=true , address=45.91.116.6:4996 , piece_id=FNI2Y3OTTLL2RTNOMJO3Q4TSPKYNIJP2WSVIPTRECKIQLZVK7IRA, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.199262158 (12h15m28.943319021s) [noise=true , address=45.91.116.11:4151 , piece_id=TFH2SWQ3CQNYO7ERZ3NBTDOY5YBBW37OXBSSLHQH4Y6T4IQGZD3Q, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.200304799 (12h15m28.944361668s) [noise=true , address=104.36.82.47:4536 , piece_id=IRJPEJBQM7KHBXOBYIE24PT5V5MLYUJJBK53WJJA54ZVCUHVVTPA, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.200901931 (12h15m28.944958817s) [noise=true , address=104.36.82.14:4252 , piece_id=3XBH5BKMWTKSDBUEWXWBDLPNB2LLAVM5CMRWU7LN5QISMCKHN7WA, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.203062990 (12h15m28.947119866s) [noise=true , address=104.36.82.46:4504 , piece_id=2KRSEO7DQGWVHJRFDXN5BLOQXNMNS7I3553TTTXY6676QZRLNV2A, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.204330863 (12h15m28.948387732s) [noise=true , address=104.36.82.62:4440 , piece_id=RCWMWE542MBKLDSI25F75FX6KK4PF5CQO43EWW4DQZTAILRLRVUQ, seg_pos={0 0}, upload=12632]: Uploading piece... Aug 7 21:55:07.759483423 (12h15m29.503540287s) [upload=12632]: Next returned nil segment Aug 7 21:55:07.759506155 (12h15m29.503563032s) [upload=12632]: Waiting for error group managing segments... Aug 7 21:55:07.914290232 (12h15m29.658347092s) [noise=true , address=104.36.82.45:4937 , piece_id=CEXRQRHANULB5PPL6ZLYHGBEAOLSEMS4VXNEM4JB4ARQY5CUNYAA, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:07.927979210 (12h15m29.672036078s) [noise=true , address=104.36.82.59:4324 , piece_id=HGM5FKLCSIZ3RTD3TFSADP2CMBRCWHZ332Y2VMJEYF2THB74MFVA, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:07.930665763 (12h15m29.674722616s) [noise=true , address=104.36.82.62:4440 , piece_id=RCWMWE542MBKLDSI25F75FX6KK4PF5CQO43EWW4DQZTAILRLRVUQ, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:07.938136247 (12h15m29.6821931s) [noise=true , address=104.36.82.37:4679 , piece_id=YRR2CJYFY6DJVE7JCAFC5OUQWODTR7UQXVGG6FXBPPUQW2TVCFHQ, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:07.939077505 (12h15m29.683134356s) [noise=true , address=104.36.82.61:4407 , piece_id=KKBMJS2DYMHPJJKF7PCJX763TT3OVTOIMA3XX2K6UOX5KW2B4E5Q, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:07.964843089 (12h15m29.708899943s) [noise=true , address=104.36.82.14:4252 , piece_id=3XBH5BKMWTKSDBUEWXWBDLPNB2LLAVM5CMRWU7LN5QISMCKHN7WA, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:07.965702408 (12h15m29.709759258s) [noise=true , address=104.36.82.60:4354 , piece_id=AE4HFMEQGKPMDR7FVC3JNM5YIZMQ6W2S5ZXBYCME2CEC7V6J5LZQ, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:07.972910256 (12h15m29.716967107s) [noise=true , address=104.36.82.46:4504 , piece_id=2KRSEO7DQGWVHJRFDXN5BLOQXNMNS7I3553TTTXY6676QZRLNV2A, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:07.988856087 (12h15m29.732912949s) [noise=true , address=45.91.116.11:4151 , piece_id=TFH2SWQ3CQNYO7ERZ3NBTDOY5YBBW37OXBSSLHQH4Y6T4IQGZD3Q, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:08.078550328 (12h15m29.822607175s) [noise=true , address=45.91.116.6:4996 , piece_id=FNI2Y3OTTLL2RTNOMJO3Q4TSPKYNIJP2WSVIPTRECKIQLZVK7IRA, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:08.384315822 (12h15m30.128372685s) [noise=true , address=104.36.82.11:4143 , piece_id=BYKKXKVMC2VEAZFQF5MOZDQGDWHJ4HQUZTR34QJMY4H4AJ42D33Q, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:08.396622672 (12h15m30.140679524s) [noise=true , address=104.36.82.9:4012 , piece_id=HSE3U5ET7TC5SLU6CENH5IUZ2SCAE2ZNYCXQHXV4VKY6LN6WTNIQ, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:08.412378225 (12h15m30.156435076s) [noise=true , address=104.36.82.23:4478 , piece_id=C56Z7JYNDZLYTNLKAVBM4FUWPFGEX6DOZVPEISDKOWE64VRARUVA, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:08.517298637 (12h15m30.261355488s) [noise=true , address=104.36.82.47:4536 , piece_id=IRJPEJBQM7KHBXOBYIE24PT5V5MLYUJJBK53WJJA54ZVCUHVVTPA, seg_pos={0 0}, upload=12632]: Done uploading piece. err: <nil> Aug 7 21:55:08.517328880 (12h15m30.261385731s) [seg_pos={0 0}, upload=12632]: Segment reached optimal threshold of 14 pieces. Aug 7 21:55:08.517600908 (12h15m30.261657758s) [noise=true , address=104.36.82.56:4233 , piece_id=33U5NWSQBMNN4GPUPNSVYFP7CXPV4BXMZMNOAAOMN6J3U62BCCBA, seg_pos={0 0}, upload=12632]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:08.517777500 (12h15m30.261834352s) [noise=true , address=79.127.205.233:4634 , piece_id=MPFJGGY2DQEUOJTI6JHAPEMWVYBU3ULOFAPHCFBDKV7KUJVZ3S2A, seg_pos={0 0}, upload=12632]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u4Nx4B4P8jBp9UxfgynkqgBbPUGVw99YAYX4UkCe9wZ1Jdd1c): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:08.517894303 (12h15m30.261951153s) [noise=true , address=104.36.82.4:4825 , piece_id=FYBWSKJSJ47XJ7YEHK3ZW5LGXNS6TWE4EWE2QUEPEAKYXAYWYOTQ, seg_pos={0 0}, upload=12632]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb): protocol: write tcp 139.59.144.22:40994->104.36.82.4:4825: use of closed network connection; write tcp 139.59.144.22:40994->104.36.82.4:4825: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40994->104.36.82.4:4825: use of closed network connection Aug 7 21:55:08.518008332 (12h15m30.262065178s) [noise=true , address=104.36.82.5:4872 , piece_id=ROROCMYX65Y2FJOOQZTHU3VI4KHVGEH3AONGULBWRMBVBAH6MWVQ, seg_pos={0 0}, upload=12632]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zqxiFpqKeXT9mhXUDuZZmvJkhejC8gcHTQgTadY1xUpQyLyZi): protocol: write tcp 139.59.144.22:56836->104.36.82.5:4872: use of closed network connection; write tcp 139.59.144.22:56836->104.36.82.5:4872: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56836->104.36.82.5:4872: use of closed network connection Aug 7 21:55:08.518025414 (12h15m30.262082281s) [seg_pos={0 0}, upload=12632]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:55:08.518110525 (12h15m30.262167386s) [upload=12632]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:55:08.518117562 (12h15m30.262174422s) [upload=12632]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:55:08.707424940 (12h15m30.451481811s) [upload=12632]: Done uploading segments... Aug 7 21:55:08.707447303 (12h15m30.45150415s) [upload=12632]: Upload finished. err: <nil> Duration:* 2.21861127s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.................] duration: 2.746181509s FAIL (2025-08-07 21:55:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.7462153s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [.....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................] duration: 875.392492ms FAIL (2025-08-07 21:54:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:41.354808211 (12h1m3.098865072s) [upload=12402]: Starting upload Aug 7 21:40:41.354938701 (12h1m3.098995568s) [upload=12402]: Done starting upload Aug 7 21:40:41.355036214 (12h1m3.099093064s) [upload=12402]: Uploading segments... Aug 7 21:40:41.355109185 (12h1m3.099166055s) [upload=12402]: Got next segment. Inline: true Aug 7 21:40:41.355116835 (12h1m3.0991737s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:40:41.355183407 (12h1m3.09924027s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:41.355190433 (12h1m3.099247297s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:41.355194357 (12h1m3.09925122s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:11.078938147 (12h1m32.822994999s) [upload=12402]: Done uploading segments... Aug 7 21:41:11.078958711 (12h1m32.823015562s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.899938086 (12h1m56.643994947s) [upload=12417]: Starting upload Aug 7 21:41:34.900014273 (12h1m56.644071133s) [upload=12417]: Done starting upload Aug 7 21:41:34.900081071 (12h1m56.644137926s) [upload=12417]: Uploading segments... Aug 7 21:41:34.900153322 (12h1m56.644210172s) [upload=12417]: Got next segment. Inline: true Aug 7 21:41:34.900158578 (12h1m56.644215438s) [upload=12417]: Waiting for error group managing segments... Aug 7 21:41:34.900228776 (12h1m56.644285623s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.900237327 (12h1m56.644294194s) [upload=12417]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.900241831 (12h1m56.644298696s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:04.627262793 (12h2m26.371319653s) [upload=12417]: Done uploading segments... Aug 7 21:42:04.627301743 (12h2m26.371358604s) [upload=12417]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:18.477725352 (12h2m40.221782248s) [upload=12428]: Starting upload Aug 7 21:42:18.477815072 (12h2m40.221871935s) [upload=12428]: Done starting upload Aug 7 21:42:18.477903321 (12h2m40.221960189s) [upload=12428]: Uploading segments... Aug 7 21:42:18.478047763 (12h2m40.222104633s) [upload=12428]: Got next segment. Inline: true Aug 7 21:42:18.478056185 (12h2m40.222113051s) [upload=12428]: Waiting for error group managing segments... Aug 7 21:42:18.478131478 (12h2m40.222188328s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:18.478138171 (12h2m40.222195021s) [upload=12428]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:18.478144895 (12h2m40.222201761s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:48.201994950 (12h3m9.946051801s) [upload=12428]: Done uploading segments... Aug 7 21:42:48.202028187 (12h3m9.946085036s) [upload=12428]: Upload finished. err: metaclient: context deadline exceeded Duration: 875.4241ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXXX......X.X..........] duration: 2.856054842s FAIL (2025-08-07 21:54:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.856084966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 858.104223ms FAIL (2025-08-07 21:54:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests Duration: 858.133583ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXX....X..............] duration: 2.68177589s FAIL (2025-08-07 21:54:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== :35.590125129 (12h5m57.33418199s) [upload=12475]: Got next segment. Inline: false Aug 7 21:45:35.590176001 (12h5m57.33423285s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:35.590181518 (12h5m57.334238369s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:35.699013188 (12h5m57.443070052s) [seg_pos={0 0}, upload=12475]: Begin upload segment... Aug 7 21:45:35.699066084 (12h5m57.443122935s) [seg_pos={0 0}, upload=12475]: Done begin upload segment. Aug 7 21:45:35.699091982 (12h5m57.443148842s) [noise=true , address=77.90.31.17:4771 , piece_id=LDTJCTYFHCPL2SHCYONX3SJWBX3GME4GGILSOCHZWG67HKEAHDYQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.699635648 (12h5m57.443692518s) [noise=true , address=104.36.82.11:4115 , piece_id=U6HOQABV6JV5T4MFFT3EDQF6R77LXTPWZ33FJWUEYOWLWLPFLYWA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700048642 (12h5m57.444105509s) [noise=true , address=104.36.82.5:4840 , piece_id=V3VLVQPAAHDACW2WGUPYAGVDU2NRGR4FLQD2SFKVP2XLQALKP3SA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700358698 (12h5m57.444415561s) [noise=true , address=45.91.116.5:4299 , piece_id=YK7OVMQRE2PYUS2LDZOZGBKYV42U6VDWI2D6CMI4D2CIEV2DBACQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700672646 (12h5m57.444729509s) [noise=true , address=45.91.116.14:4191 , piece_id=TYBU2DWWTVWWEEMTBD2PIQKVIU3YRHKXH7ACD7Y3VKZVDXPDRALA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701028296 (12h5m57.445085159s) [noise=true , address=104.36.82.13:4187 , piece_id=WCIN3NV6AFBYDUGO7TDPOMZG5YJ24ISXVB2WSLB46EZOPPGQKW2Q, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701400580 (12h5m57.44545744s) [noise=true , address=77.90.31.15:4697 , piece_id=XQ5CZFWWVUZ6MKSIYZVOUZG6YAA7UAVVISSKVAPJXXD7HPXJHK3A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701701900 (12h5m57.445758763s) [noise=true , address=45.91.116.11:4119 , piece_id=K5FKXMYXUWHRKFZPDHFKZ425RAZLEQH254K65QOITKTC4APUYL7A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.702301822 (12h5m57.446358688s) [noise=true , address=45.91.116.11:4118 , piece_id=QPX3R57IYGZFNNOCPHOSQCRRQH6KJMDYSBPGHLX34DNABYYUXZ6A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.702831318 (12h5m57.446888179s) [noise=true , address=104.36.82.10:4078 , piece_id=BEMP3VUCCLIJAY2OU2TFDJ5SQDJFO4OGL3UEHQ7YPMJGBRZ4E6BQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.703272004 (12h5m57.447328915s) [noise=true , address=45.91.116.6:4974 , piece_id=5HPCLRLKSQWUBMWQXW5LEOVQVBIKV62QAYF4YKQGQREWT3ZTAI2Q, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.703720399 (12h5m57.447777259s) [noise=true , address=104.36.82.3:4767 , piece_id=D3DBXBQ3UNQCWF5DR3HV46XZSOMRQHR33SQ6VSA7BXY7OZEWXNAA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704065340 (12h5m57.44812221s) [noise=true , address=77.90.31.13:4626 , piece_id=KM7OLI4GIFFIK4UTVRX26KMGVRZMASK47B7B7FOYCS6DU5SMZEXA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704350352 (12h5m57.448407212s) [noise=true , address=104.36.82.12:4151 , piece_id=UUTIELJOX7CUEFHHFWDD4XETR3EB7IQOCKUZXQALKKGXPBVWH4TA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704707337 (12h5m57.448764198s) [noise=true , address=77.90.31.12:4590 , piece_id=BF63RJ2G676UQFZGWIAXFCZR2DC67F4AH5VZAB3MYIDDOMJKGISA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.705111011 (12h5m57.449167878s) [noise=true , address=104.36.82.7:4911 , piece_id=6HSMRZBG26Y2XZ7NQZ6JD7P2BXZTDFDC6A5JIIS7SM7T6FAL2PWQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:36.332909886 (12h5m58.076966752s) [upload=12475]: Next returned nil segment Aug 7 21:45:36.332933437 (12h5m58.076990311s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:45:36.489772849 (12h5m58.233829715s) [noise=true , address=104.36.82.10:4078 , piece_id=BEMP3VUCCLIJAY2OU2TFDJ5SQDJFO4OGL3UEHQ7YPMJGBRZ4E6BQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.509052197 (12h5m58.253109052s) [noise=true , address=104.36.82.12:4151 , piece_id=UUTIELJOX7CUEFHHFWDD4XETR3EB7IQOCKUZXQALKKGXPBVWH4TA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.542228341 (12h5m58.286285191s) [noise=true , address=77.90.31.12:4590 , piece_id=BF63RJ2G676UQFZGWIAXFCZR2DC67F4AH5VZAB3MYIDDOMJKGISA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.573060953 (12h5m58.317117805s) [noise=true , address=45.91.116.6:4974 , piece_id=5HPCLRLKSQWUBMWQXW5LEOVQVBIKV62QAYF4YKQGQREWT3ZTAI2Q, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.573821057 (12h5m58.317877923s) [noise=true , address=45.91.116.5:4299 , piece_id=YK7OVMQRE2PYUS2LDZOZGBKYV42U6VDWI2D6CMI4D2CIEV2DBACQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.620961481 (12h5m58.365018331s) [noise=true , address=77.90.31.17:4771 , piece_id=LDTJCTYFHCPL2SHCYONX3SJWBX3GME4GGILSOCHZWG67HKEAHDYQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.631043610 (12h5m58.375100483s) [noise=true , address=45.91.116.14:4191 , piece_id=TYBU2DWWTVWWEEMTBD2PIQKVIU3YRHKXH7ACD7Y3VKZVDXPDRALA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.632954530 (12h5m58.377011377s) [noise=true , address=45.91.116.11:4119 , piece_id=K5FKXMYXUWHRKFZPDHFKZ425RAZLEQH254K65QOITKTC4APUYL7A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.644557641 (12h5m58.388614491s) [noise=true , address=104.36.82.7:4911 , piece_id=6HSMRZBG26Y2XZ7NQZ6JD7P2BXZTDFDC6A5JIIS7SM7T6FAL2PWQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.649884441 (12h5m58.393941291s) [noise=true , address=77.90.31.13:4626 , piece_id=KM7OLI4GIFFIK4UTVRX26KMGVRZMASK47B7B7FOYCS6DU5SMZEXA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.661770778 (12h5m58.405827645s) [noise=true , address=77.90.31.15:4697 , piece_id=XQ5CZFWWVUZ6MKSIYZVOUZG6YAA7UAVVISSKVAPJXXD7HPXJHK3A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.725730770 (12h5m58.46978762s) [noise=true , address=104.36.82.3:4767 , piece_id=D3DBXBQ3UNQCWF5DR3HV46XZSOMRQHR33SQ6VSA7BXY7OZEWXNAA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.869176760 (12h5m58.613233628s) [noise=true , address=45.91.116.11:4118 , piece_id=QPX3R57IYGZFNNOCPHOSQCRRQH6KJMDYSBPGHLX34DNABYYUXZ6A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.869206102 (12h5m58.61326296s) [seg_pos={0 0}, upload=12475]: Segment reached optimal threshold of 13 pieces. Aug 7 21:45:36.869561423 (12h5m58.613618286s) [noise=true , address=104.36.82.11:4115 , piece_id=U6HOQABV6JV5T4MFFT3EDQF6R77LXTPWZ33FJWUEYOWLWLPFLYWA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Aug 7 21:45:36.869806552 (12h5m58.613863414s) [noise=true , address=104.36.82.5:4840 , piece_id=V3VLVQPAAHDACW2WGUPYAGVDU2NRGR4FLQD2SFKVP2XLQALKP3SA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection; write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection Aug 7 21:45:36.869995683 (12h5m58.614052547s) [noise=true , address=104.36.82.13:4187 , piece_id=WCIN3NV6AFBYDUGO7TDPOMZG5YJ24ISXVB2WSLB46EZOPPGQKW2Q, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection Aug 7 21:45:36.870032790 (12h5m58.614089658s) [seg_pos={0 0}, upload=12475]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:45:36.870148681 (12h5m58.614205541s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:45:36.870154041 (12h5m58.614210892s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:37.015978570 (12h5m58.76003542s) [upload=12475]: Done uploading segments... Aug 7 21:45:37.016019019 (12h5m58.760075869s) [upload=12475]: Upload finished. err: <nil> Aug 7 21:46:05.392535892 (12h6m27.136592742s) [upload=12486]: Starting upload Aug 7 21:46:05.392632482 (12h6m27.136689348s) [upload=12486]: Done starting upload Aug 7 21:46:05.392784466 (12h6m27.136841327s) [upload=12486]: Uploading segments... Aug 7 21:46:05.392963140 (12h6m27.13701999s) [upload=12486]: Got next segment. Inline: false Aug 7 21:46:05.393017249 (12h6m27.137074109s) [upload=12486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:05.393023723 (12h6m27.137080582s) [upload=12486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:46:35.116242044 (12h6m56.860298909s) [upload=12486]: Next segment err: context deadline exceeded Aug 7 21:46:35.116407347 (12h6m56.860464196s) [upload=12486]: Done uploading segments... Aug 7 21:46:35.116450315 (12h6m56.860507172s) [upload=12486]: Upload finished. err: context deadline exceeded Duration: 2.681817888s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [...................X............X.X.X.......................] duration: 110.593118ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:54:38 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 110.619902ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.................] duration: 1.088209038s FAIL (2025-08-07 21:54:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.088238159s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XXXX.........] duration: 3.063125244s FAIL (2025-08-07 21:54:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== tchRequestItem_ObjectBegin Aug 7 21:24:45.813454815 (11h45m7.557511676s) [upload=12171]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:43.107142097 (11h46m4.851198968s) [seg_pos={0 0}, upload=12171]: Begin upload segment... Aug 7 21:25:43.107205391 (11h46m4.851262241s) [seg_pos={0 0}, upload=12171]: Done begin upload segment. Aug 7 21:25:43.107237087 (11h46m4.851293957s) [noise=true , address=104.36.82.15:4269 , piece_id=HTXIUN5HOU4B66MPMEXJJSUL6JNBE6FI6EB62IEG7P3WTZLAH47Q, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.107267905 (11h46m4.851324768s) [noise=true , address=104.36.82.8:4983 , piece_id=ZDR4NMXM2MJTWTLT5776GQIUXCIY7TMRNBQ43L73XOFFBLSA6KQQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.107830175 (11h46m4.851887046s) [noise=true , address=45.91.116.7:4013 , piece_id=JEWYLXASDWIZ6G3LSHWMUAFAZIKLWYO7KOZLEQEGNZB3HMNVSKYA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.108780038 (11h46m4.852836904s) [noise=true , address=104.36.82.16:4319 , piece_id=36CAJ2JJZF3CLN67PLA7VUM2F6NULTFSQ3WG6EDE734G3LNFZJTA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.108959605 (11h46m4.853016472s) [noise=true , address=104.36.82.43:4852 , piece_id=NBGNTQUWEOVY72D3XFFO74EFUL52BZP222Z6FNIYCFB43FOBIW2Q, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.109220359 (11h46m4.853277218s) [noise=true , address=79.127.201.214:4296 , piece_id=SK5JKFXZVF5XRW6STC5KDH5WNI7JHB6X5X5MLZ6WLLHG5SO6APSA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.109377214 (11h46m4.853434079s) [noise=true , address=104.36.82.37:4662 , piece_id=KZG6JYEDQUTQGQCTUAZK7R3JRD7GKFLKORVEC5LIZ47GVJ6MFZRA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.109626289 (11h46m4.853683159s) [noise=true , address=104.36.82.13:4195 , piece_id=YVPFOIDCG5F4LET7V5SZU2X6WKD4MCZWIQ6HUE7NVIOIVBRGDNHA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.109743295 (11h46m4.853800166s) [noise=true , address=79.127.205.245:4234 , piece_id=RADEPAKA2ZEQSI5UUX5BHRAR5JHNMOQI7MGZW42A4ZW52KJOXBRA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.110014430 (11h46m4.854071298s) [noise=true , address=64.49.13.138:4487 , piece_id=MXQL4B6PL3FOQS4WCCGCPF5DT5C3JDLYJNOEIE7GKNE5FRQJZ7EA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.110156657 (11h46m4.854213527s) [noise=true , address=45.91.116.8:4056 , piece_id=DOFJMEXQQBOTVMCWC77YLLDCUY4XYYRM3X5BOBSXXAPB7T4YVHZQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.110413863 (11h46m4.854470734s) [noise=true , address=104.36.82.35:4607 , piece_id=DQKF7V3EW6OT5C4IFITEQNOXUUHDOMG5AESWXUYJP3SERMWUF5VQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.110778464 (11h46m4.854835327s) [noise=true , address=104.36.82.38:4708 , piece_id=JUSE5AFFHKPHCDVDRN5WWWXQK6VHIY3CXQWMJ4EXEJXVMICMZIXA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.111117136 (11h46m4.855173999s) [noise=true , address=45.91.116.14:4197 , piece_id=52LI7AMEDE2OBW6TAIOP5Y2M4J4HAWEITEKIYPZZ7XQ2Z5CGYIBQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.111560812 (11h46m4.855617682s) [noise=true , address=104.36.82.35:4602 , piece_id=42ZEPU7YI52FBKWCUG2ZRDLPOLG5IYXSPR4HJ2QUSURO65WIAICQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.111976240 (11h46m4.856033107s) [noise=true , address=104.36.82.51:4073 , piece_id=HPC5WAIPHCOMTZDKPDPGZ7UJ43KPZZ2IFL5VFH7ILXHCF63D5NXA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.112141257 (11h46m4.856198127s) [noise=true , address=104.36.82.22:4511 , piece_id=WWW6R3JFLEQCGIAIZU6VXOC6XCPXFC3AWJ74PBL4PQXKTTSF535A, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.112560277 (11h46m4.856617148s) [noise=true , address=104.36.82.62:4412 , piece_id=6NAJYYYU3AX7KJNFBXTPEYRCFFTMV3ESLVFUWTEGKGNOEHFHJJKA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.675139110 (11h46m5.419195972s) [upload=12171]: Next returned nil segment Aug 7 21:25:43.675161049 (11h46m5.419217914s) [upload=12171]: Waiting for error group managing segments... Aug 7 21:25:43.872519019 (11h46m5.616575886s) [noise=true , address=104.36.82.62:4412 , piece_id=6NAJYYYU3AX7KJNFBXTPEYRCFFTMV3ESLVFUWTEGKGNOEHFHJJKA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.876126423 (11h46m5.62018327s) [noise=true , address=104.36.82.51:4073 , piece_id=HPC5WAIPHCOMTZDKPDPGZ7UJ43KPZZ2IFL5VFH7ILXHCF63D5NXA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.882439205 (11h46m5.626496076s) [noise=true , address=104.36.82.35:4602 , piece_id=42ZEPU7YI52FBKWCUG2ZRDLPOLG5IYXSPR4HJ2QUSURO65WIAICQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.886290836 (11h46m5.63034769s) [noise=true , address=104.36.82.22:4511 , piece_id=WWW6R3JFLEQCGIAIZU6VXOC6XCPXFC3AWJ74PBL4PQXKTTSF535A, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.888219892 (11h46m5.632276738s) [noise=true , address=104.36.82.15:4269 , piece_id=HTXIUN5HOU4B66MPMEXJJSUL6JNBE6FI6EB62IEG7P3WTZLAH47Q, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.888421317 (11h46m5.632478185s) [noise=true , address=45.91.116.14:4197 , piece_id=52LI7AMEDE2OBW6TAIOP5Y2M4J4HAWEITEKIYPZZ7XQ2Z5CGYIBQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.889366258 (11h46m5.633423104s) [noise=true , address=104.36.82.35:4607 , piece_id=DQKF7V3EW6OT5C4IFITEQNOXUUHDOMG5AESWXUYJP3SERMWUF5VQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.890288566 (11h46m5.634345437s) [noise=true , address=104.36.82.43:4852 , piece_id=NBGNTQUWEOVY72D3XFFO74EFUL52BZP222Z6FNIYCFB43FOBIW2Q, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.893016005 (11h46m5.637072875s) [noise=true , address=45.91.116.8:4056 , piece_id=DOFJMEXQQBOTVMCWC77YLLDCUY4XYYRM3X5BOBSXXAPB7T4YVHZQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.903527905 (11h46m5.647584771s) [noise=true , address=104.36.82.38:4708 , piece_id=JUSE5AFFHKPHCDVDRN5WWWXQK6VHIY3CXQWMJ4EXEJXVMICMZIXA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.912802006 (11h46m5.656858856s) [noise=true , address=45.91.116.7:4013 , piece_id=JEWYLXASDWIZ6G3LSHWMUAFAZIKLWYO7KOZLEQEGNZB3HMNVSKYA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.974200245 (11h46m5.718257092s) [noise=true , address=104.36.82.8:4983 , piece_id=ZDR4NMXM2MJTWTLT5776GQIUXCIY7TMRNBQ43L73XOFFBLSA6KQQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:44.128893332 (11h46m5.872950195s) [noise=true , address=79.127.201.214:4296 , piece_id=SK5JKFXZVF5XRW6STC5KDH5WNI7JHB6X5X5MLZ6WLLHG5SO6APSA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:44.143902251 (11h46m5.887959098s) [noise=true , address=104.36.82.37:4662 , piece_id=KZG6JYEDQUTQGQCTUAZK7R3JRD7GKFLKORVEC5LIZ47GVJ6MFZRA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:44.143991645 (11h46m5.888048495s) [seg_pos={0 0}, upload=12171]: Segment reached optimal threshold of 14 pieces. Aug 7 21:25:44.144122598 (11h46m5.888179458s) [noise=true , address=64.49.13.138:4487 , piece_id=MXQL4B6PL3FOQS4WCCGCPF5DT5C3JDLYJNOEIE7GKNE5FRQJZ7EA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF Aug 7 21:25:44.144194345 (11h46m5.888251205s) [noise=true , address=79.127.205.245:4234 , piece_id=RADEPAKA2ZEQSI5UUX5BHRAR5JHNMOQI7MGZW42A4ZW52KJOXBRA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJWq44d6y4Ai4tyZpCUvCgZeMHrVdiiSSngTR75YZZXbsgq2d): protocol: expected piece hash; context canceled; EOF Aug 7 21:25:44.144213145 (11h46m5.888269992s) [noise=true , address=104.36.82.13:4195 , piece_id=YVPFOIDCG5F4LET7V5SZU2X6WKD4MCZWIQ6HUE7NVIOIVBRGDNHA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH): protocol: write tcp 139.59.144.22:41808->104.36.82.13:4195: use of closed network connection; write tcp 139.59.144.22:41808->104.36.82.13:4195: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41808->104.36.82.13:4195: use of closed network connection Aug 7 21:25:44.144301767 (11h46m5.888358613s) [noise=true , address=104.36.82.16:4319 , piece_id=36CAJ2JJZF3CLN67PLA7VUM2F6NULTFSQ3WG6EDE734G3LNFZJTA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hQTG2DzecaoN1Na2mY3JJqNj2qxtJDQkhMYmRmd9hve8D7T9C): protocol: write tcp 139.59.144.22:51894->104.36.82.16:4319: use of closed network connection; write tcp 139.59.144.22:51894->104.36.82.16:4319: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51894->104.36.82.16:4319: use of closed network connection Aug 7 21:25:44.144323393 (11h46m5.88838024s) [seg_pos={0 0}, upload=12171]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:25:44.144394152 (11h46m5.888450998s) [upload=12171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:25:44.144398730 (11h46m5.888455591s) [upload=12171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:47.207768845 (12h9m8.951825696s) [upload=12171]: Done uploading segments... Aug 7 21:48:47.207799297 (12h9m8.951856149s) [upload=12171]: Upload finished. err: metaclient: context deadline exceeded Duration:* 3.063160713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X.......................XX.X..........X.....X.........] duration: 3.055849124s FAIL (2025-08-07 21:54:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== piece_id=RXXWJ22ZFGJ42YRFAX4HRE7E2VPGAKWKOEBWWTBBEKXPLRQFF54A, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.931857584 (12h7m22.675914448s) [noise=true , address=104.36.82.39:4754 , piece_id=B64XNARGUDMJTQHTSEYX6LYDVYEPSFESYSHFJ2YATHOWE7V3JTSA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.932258428 (12h7m22.676315289s) [noise=true , address=104.36.82.55:4224 , piece_id=DVH3QKF4OWIMYY4EEMSNOIXW5CC54KN2M54CHUTU6YFC5MHRQFOA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.933368803 (12h7m22.677425673s) [noise=true , address=45.91.116.9:4288 , piece_id=ILYKFA6LDTNRBRGOVOR25H2VIGOUHQZQ3WPZDV3DZMZKH6ENKVNA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.933961999 (12h7m22.678018845s) [noise=true , address=104.36.82.43:4853 , piece_id=WBEOH7537PO55GJLYGC4YHMTKULN2T6BCCYAEVBRR5X4AQ5F4XSA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.934101417 (12h7m22.67815828s) [noise=true , address=104.36.82.35:4601 , piece_id=FIWLJE2E4X22266LQ44U7IBBZ4D7XLUGN6LRGENTOTC7H5WAIKMQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.934371093 (12h7m22.678427954s) [noise=true , address=104.36.82.44:4877 , piece_id=EDM3CTKQYKOSOP3JU3UUWN63LVG7BA6FGOAFF2IRKRNRUPK7GTVQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.934756333 (12h7m22.678813196s) [noise=true , address=104.36.82.61:4405 , piece_id=BA25J7JHA2KUSFTJDRPBPBHFVIUPTD7FE7IKSTJUEU3IGBMIB72A, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.934888956 (12h7m22.678945807s) [noise=true , address=64.49.13.154:4926 , piece_id=X5CLC7JS7PCLNSPLHTDA7VC4BXKIAZON3A2EVZCBZ3NZYYZD42HA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.935299805 (12h7m22.679356668s) [noise=true , address=104.36.82.9:4027 , piece_id=R3U7T2YCU7EKQULIM6KKZYBWQP2ZGNLHUUWQ7B4KJUJJIRAPZEPA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.935539944 (12h7m22.679596796s) [noise=true , address=64.49.13.169:4708 , piece_id=HTW7Y2VQQF5VDJPPLE265SZ7JU24T2KSEIUCSPEFQKZA2HNI2QWQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.935895434 (12h7m22.679952289s) [noise=true , address=104.36.82.36:4640 , piece_id=KCDKD2VKLEHD6V52CQBLYI4KRNDDMLDGGYBQPLX27JBDAIV64XQQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.936414259 (12h7m22.680471112s) [noise=true , address=104.36.82.13:4214 , piece_id=4X2ZSNKGKTJ4OZWJVFDYTR2RBRPOJRSOOVTXSBUPCIO3K53U5HJQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.936836219 (12h7m22.680893083s) [noise=true , address=104.36.82.62:4431 , piece_id=CRGUBEWHDRPMKGY47AYWHGCX4D3RKCP5OW6UV6TD77DIIVNBVZBA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.937535515 (12h7m22.68159243s) [noise=true , address=79.127.205.233:4642 , piece_id=YGV6RGALGLVITSWGUWQFJVQNCPHXWS67G33QWZZWAPIFE6RYQQSA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.937642268 (12h7m22.681699132s) [noise=true , address=104.36.82.58:4283 , piece_id=AEQQA4INRDBRLNAYRVDD4PSLOCBFM5HP4GOCWFOX47AHHJAEBWZA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.938008889 (12h7m22.682065754s) [noise=true , address=104.36.82.37:4688 , piece_id=LBVMHMZJFVI5HD3P33LOA2L5KA7EUB64TQWVFX3PSYUUSLCCLAPA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.938565671 (12h7m22.682622538s) [noise=true , address=104.36.82.23:4452 , piece_id=4XKVMVVLGCJRPKBC5HVJSX6M6UUI4QGFOCCKRHCMPNIOI3ULLWSA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:01.515934007 (12h7m23.259990877s) [upload=12498]: Next returned nil segment Aug 7 21:47:01.515957554 (12h7m23.260014424s) [upload=12498]: Waiting for error group managing segments... Aug 7 21:47:01.689221575 (12h7m23.433278437s) [noise=true , address=104.36.82.58:4283 , piece_id=AEQQA4INRDBRLNAYRVDD4PSLOCBFM5HP4GOCWFOX47AHHJAEBWZA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.696245173 (12h7m23.440302028s) [noise=true , address=104.36.82.45:4931 , piece_id=RXXWJ22ZFGJ42YRFAX4HRE7E2VPGAKWKOEBWWTBBEKXPLRQFF54A, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.698487716 (12h7m23.442544583s) [noise=true , address=104.36.82.23:4452 , piece_id=4XKVMVVLGCJRPKBC5HVJSX6M6UUI4QGFOCCKRHCMPNIOI3ULLWSA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.699902328 (12h7m23.443959181s) [noise=true , address=104.36.82.55:4224 , piece_id=DVH3QKF4OWIMYY4EEMSNOIXW5CC54KN2M54CHUTU6YFC5MHRQFOA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.704569768 (12h7m23.448626616s) [noise=true , address=104.36.82.62:4431 , piece_id=CRGUBEWHDRPMKGY47AYWHGCX4D3RKCP5OW6UV6TD77DIIVNBVZBA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.713265789 (12h7m23.457322661s) [noise=true , address=104.36.82.39:4754 , piece_id=B64XNARGUDMJTQHTSEYX6LYDVYEPSFESYSHFJ2YATHOWE7V3JTSA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.733750309 (12h7m23.477807164s) [noise=true , address=104.36.82.36:4640 , piece_id=KCDKD2VKLEHD6V52CQBLYI4KRNDDMLDGGYBQPLX27JBDAIV64XQQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.746947712 (12h7m23.491004568s) [noise=true , address=104.36.82.13:4214 , piece_id=4X2ZSNKGKTJ4OZWJVFDYTR2RBRPOJRSOOVTXSBUPCIO3K53U5HJQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.755976189 (12h7m23.500033045s) [noise=true , address=104.36.82.43:4853 , piece_id=WBEOH7537PO55GJLYGC4YHMTKULN2T6BCCYAEVBRR5X4AQ5F4XSA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.795757727 (12h7m23.539814619s) [noise=true , address=104.36.82.9:4027 , piece_id=R3U7T2YCU7EKQULIM6KKZYBWQP2ZGNLHUUWQ7B4KJUJJIRAPZEPA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.901256341 (12h7m23.645313207s) [noise=true , address=104.36.82.44:4877 , piece_id=EDM3CTKQYKOSOP3JU3UUWN63LVG7BA6FGOAFF2IRKRNRUPK7GTVQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.922821824 (12h7m23.666878693s) [noise=true , address=45.91.116.9:4288 , piece_id=ILYKFA6LDTNRBRGOVOR25H2VIGOUHQZQ3WPZDV3DZMZKH6ENKVNA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.950282380 (12h7m23.694339244s) [noise=true , address=104.36.82.37:4688 , piece_id=LBVMHMZJFVI5HD3P33LOA2L5KA7EUB64TQWVFX3PSYUUSLCCLAPA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:02.145128156 (12h7m23.889185029s) [noise=true , address=104.36.82.35:4601 , piece_id=FIWLJE2E4X22266LQ44U7IBBZ4D7XLUGN6LRGENTOTC7H5WAIKMQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:02.145174967 (12h7m23.889231834s) [seg_pos={0 0}, upload=12498]: Segment reached optimal threshold of 14 pieces. Aug 7 21:47:02.145375024 (12h7m23.889431872s) [noise=true , address=79.127.205.233:4642 , piece_id=YGV6RGALGLVITSWGUWQFJVQNCPHXWS67G33QWZZWAPIFE6RYQQSA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MYs8DF38z84JzwWUYRL7rxPZWNxfyH4WnUdT1L58BiLkQsUTW): protocol: expected piece hash; context canceled; EOF Aug 7 21:47:02.145801827 (12h7m23.889858706s) [noise=true , address=64.49.13.154:4926 , piece_id=X5CLC7JS7PCLNSPLHTDA7VC4BXKIAZON3A2EVZCBZ3NZYYZD42HA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bpEYLHxV7TM3KdheTwmwBxpPvm3PKccMVFLfLc5fTXtJZX5VY): protocol: expected piece hash; context canceled; EOF Aug 7 21:47:02.146021528 (12h7m23.890078399s) [noise=true , address=64.49.13.169:4708 , piece_id=HTW7Y2VQQF5VDJPPLE265SZ7JU24T2KSEIUCSPEFQKZA2HNI2QWQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: expected piece hash; context canceled; EOF Aug 7 21:47:02.146292773 (12h7m23.890349638s) [noise=true , address=104.36.82.61:4405 , piece_id=BA25J7JHA2KUSFTJDRPBPBHFVIUPTD7FE7IKSTJUEU3IGBMIB72A, seg_pos={0 0}, upload=12498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XPFXQaLxDzhZjcaACurTejzdA8S4szrXVJjc5pgJm9UMreNiy): protocol: read tcp 139.59.144.22:49992->104.36.82.61:4405: use of closed network connection; context canceled Aug 7 21:47:02.146329902 (12h7m23.890386769s) [seg_pos={0 0}, upload=12498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:47:02.146430465 (12h7m23.890487825s) [upload=12498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:47:02.146437380 (12h7m23.890494243s) [upload=12498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:02.446739950 (12h7m24.190796819s) [upload=12498]: Done uploading segments... Aug 7 21:47:02.446788220 (12h7m24.190845097s) [upload=12498]: Upload finished. err: <nil> Aug 7 21:47:52.318171063 (12h8m14.06222791s) [upload=12512]: Starting upload Aug 7 21:47:52.318240558 (12h8m14.062297427s) [upload=12512]: Done starting upload Aug 7 21:47:52.318352095 (12h8m14.062408942s) [upload=12512]: Uploading segments... Aug 7 21:47:52.318602540 (12h8m14.062659412s) [upload=12512]: Got next segment. Inline: false Aug 7 21:47:52.318662425 (12h8m14.062719297s) [upload=12512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:52.318669510 (12h8m14.062726381s) [upload=12512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:47:52.406892213 (12h8m14.150949079s) [upload=12512]: Next segment err: metaclient: Too Many Requests Aug 7 21:47:52.406918034 (12h8m14.150974895s) [upload=12512]: Done uploading segments... Aug 7 21:47:52.406934956 (12h8m14.150991807s) [upload=12512]: Upload finished. err: metaclient: Too Many Requests Duration: 3.055879816s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 29 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......] duration: 2.268048786s FAIL (2025-08-07 21:54:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.268078149s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 5 out of 60 is failed (threshold=3) [...................XX..............XX.X.....................] duration: 310.441883ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:54:20 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 310.456797ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXX....X.............] duration: 2.683069841s FAIL (2025-08-07 21:54:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== :35.590125129 (12h5m57.33418199s) [upload=12475]: Got next segment. Inline: false Aug 7 21:45:35.590176001 (12h5m57.33423285s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:35.590181518 (12h5m57.334238369s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:35.699013188 (12h5m57.443070052s) [seg_pos={0 0}, upload=12475]: Begin upload segment... Aug 7 21:45:35.699066084 (12h5m57.443122935s) [seg_pos={0 0}, upload=12475]: Done begin upload segment. Aug 7 21:45:35.699091982 (12h5m57.443148842s) [noise=true , address=77.90.31.17:4771 , piece_id=LDTJCTYFHCPL2SHCYONX3SJWBX3GME4GGILSOCHZWG67HKEAHDYQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.699635648 (12h5m57.443692518s) [noise=true , address=104.36.82.11:4115 , piece_id=U6HOQABV6JV5T4MFFT3EDQF6R77LXTPWZ33FJWUEYOWLWLPFLYWA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700048642 (12h5m57.444105509s) [noise=true , address=104.36.82.5:4840 , piece_id=V3VLVQPAAHDACW2WGUPYAGVDU2NRGR4FLQD2SFKVP2XLQALKP3SA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700358698 (12h5m57.444415561s) [noise=true , address=45.91.116.5:4299 , piece_id=YK7OVMQRE2PYUS2LDZOZGBKYV42U6VDWI2D6CMI4D2CIEV2DBACQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700672646 (12h5m57.444729509s) [noise=true , address=45.91.116.14:4191 , piece_id=TYBU2DWWTVWWEEMTBD2PIQKVIU3YRHKXH7ACD7Y3VKZVDXPDRALA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701028296 (12h5m57.445085159s) [noise=true , address=104.36.82.13:4187 , piece_id=WCIN3NV6AFBYDUGO7TDPOMZG5YJ24ISXVB2WSLB46EZOPPGQKW2Q, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701400580 (12h5m57.44545744s) [noise=true , address=77.90.31.15:4697 , piece_id=XQ5CZFWWVUZ6MKSIYZVOUZG6YAA7UAVVISSKVAPJXXD7HPXJHK3A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701701900 (12h5m57.445758763s) [noise=true , address=45.91.116.11:4119 , piece_id=K5FKXMYXUWHRKFZPDHFKZ425RAZLEQH254K65QOITKTC4APUYL7A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.702301822 (12h5m57.446358688s) [noise=true , address=45.91.116.11:4118 , piece_id=QPX3R57IYGZFNNOCPHOSQCRRQH6KJMDYSBPGHLX34DNABYYUXZ6A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.702831318 (12h5m57.446888179s) [noise=true , address=104.36.82.10:4078 , piece_id=BEMP3VUCCLIJAY2OU2TFDJ5SQDJFO4OGL3UEHQ7YPMJGBRZ4E6BQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.703272004 (12h5m57.447328915s) [noise=true , address=45.91.116.6:4974 , piece_id=5HPCLRLKSQWUBMWQXW5LEOVQVBIKV62QAYF4YKQGQREWT3ZTAI2Q, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.703720399 (12h5m57.447777259s) [noise=true , address=104.36.82.3:4767 , piece_id=D3DBXBQ3UNQCWF5DR3HV46XZSOMRQHR33SQ6VSA7BXY7OZEWXNAA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704065340 (12h5m57.44812221s) [noise=true , address=77.90.31.13:4626 , piece_id=KM7OLI4GIFFIK4UTVRX26KMGVRZMASK47B7B7FOYCS6DU5SMZEXA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704350352 (12h5m57.448407212s) [noise=true , address=104.36.82.12:4151 , piece_id=UUTIELJOX7CUEFHHFWDD4XETR3EB7IQOCKUZXQALKKGXPBVWH4TA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704707337 (12h5m57.448764198s) [noise=true , address=77.90.31.12:4590 , piece_id=BF63RJ2G676UQFZGWIAXFCZR2DC67F4AH5VZAB3MYIDDOMJKGISA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.705111011 (12h5m57.449167878s) [noise=true , address=104.36.82.7:4911 , piece_id=6HSMRZBG26Y2XZ7NQZ6JD7P2BXZTDFDC6A5JIIS7SM7T6FAL2PWQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:36.332909886 (12h5m58.076966752s) [upload=12475]: Next returned nil segment Aug 7 21:45:36.332933437 (12h5m58.076990311s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:45:36.489772849 (12h5m58.233829715s) [noise=true , address=104.36.82.10:4078 , piece_id=BEMP3VUCCLIJAY2OU2TFDJ5SQDJFO4OGL3UEHQ7YPMJGBRZ4E6BQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.509052197 (12h5m58.253109052s) [noise=true , address=104.36.82.12:4151 , piece_id=UUTIELJOX7CUEFHHFWDD4XETR3EB7IQOCKUZXQALKKGXPBVWH4TA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.542228341 (12h5m58.286285191s) [noise=true , address=77.90.31.12:4590 , piece_id=BF63RJ2G676UQFZGWIAXFCZR2DC67F4AH5VZAB3MYIDDOMJKGISA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.573060953 (12h5m58.317117805s) [noise=true , address=45.91.116.6:4974 , piece_id=5HPCLRLKSQWUBMWQXW5LEOVQVBIKV62QAYF4YKQGQREWT3ZTAI2Q, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.573821057 (12h5m58.317877923s) [noise=true , address=45.91.116.5:4299 , piece_id=YK7OVMQRE2PYUS2LDZOZGBKYV42U6VDWI2D6CMI4D2CIEV2DBACQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.620961481 (12h5m58.365018331s) [noise=true , address=77.90.31.17:4771 , piece_id=LDTJCTYFHCPL2SHCYONX3SJWBX3GME4GGILSOCHZWG67HKEAHDYQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.631043610 (12h5m58.375100483s) [noise=true , address=45.91.116.14:4191 , piece_id=TYBU2DWWTVWWEEMTBD2PIQKVIU3YRHKXH7ACD7Y3VKZVDXPDRALA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.632954530 (12h5m58.377011377s) [noise=true , address=45.91.116.11:4119 , piece_id=K5FKXMYXUWHRKFZPDHFKZ425RAZLEQH254K65QOITKTC4APUYL7A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.644557641 (12h5m58.388614491s) [noise=true , address=104.36.82.7:4911 , piece_id=6HSMRZBG26Y2XZ7NQZ6JD7P2BXZTDFDC6A5JIIS7SM7T6FAL2PWQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.649884441 (12h5m58.393941291s) [noise=true , address=77.90.31.13:4626 , piece_id=KM7OLI4GIFFIK4UTVRX26KMGVRZMASK47B7B7FOYCS6DU5SMZEXA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.661770778 (12h5m58.405827645s) [noise=true , address=77.90.31.15:4697 , piece_id=XQ5CZFWWVUZ6MKSIYZVOUZG6YAA7UAVVISSKVAPJXXD7HPXJHK3A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.725730770 (12h5m58.46978762s) [noise=true , address=104.36.82.3:4767 , piece_id=D3DBXBQ3UNQCWF5DR3HV46XZSOMRQHR33SQ6VSA7BXY7OZEWXNAA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.869176760 (12h5m58.613233628s) [noise=true , address=45.91.116.11:4118 , piece_id=QPX3R57IYGZFNNOCPHOSQCRRQH6KJMDYSBPGHLX34DNABYYUXZ6A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.869206102 (12h5m58.61326296s) [seg_pos={0 0}, upload=12475]: Segment reached optimal threshold of 13 pieces. Aug 7 21:45:36.869561423 (12h5m58.613618286s) [noise=true , address=104.36.82.11:4115 , piece_id=U6HOQABV6JV5T4MFFT3EDQF6R77LXTPWZ33FJWUEYOWLWLPFLYWA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Aug 7 21:45:36.869806552 (12h5m58.613863414s) [noise=true , address=104.36.82.5:4840 , piece_id=V3VLVQPAAHDACW2WGUPYAGVDU2NRGR4FLQD2SFKVP2XLQALKP3SA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection; write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection Aug 7 21:45:36.869995683 (12h5m58.614052547s) [noise=true , address=104.36.82.13:4187 , piece_id=WCIN3NV6AFBYDUGO7TDPOMZG5YJ24ISXVB2WSLB46EZOPPGQKW2Q, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection Aug 7 21:45:36.870032790 (12h5m58.614089658s) [seg_pos={0 0}, upload=12475]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:45:36.870148681 (12h5m58.614205541s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:45:36.870154041 (12h5m58.614210892s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:37.015978570 (12h5m58.76003542s) [upload=12475]: Done uploading segments... Aug 7 21:45:37.016019019 (12h5m58.760075869s) [upload=12475]: Upload finished. err: <nil> Aug 7 21:46:05.392535892 (12h6m27.136592742s) [upload=12486]: Starting upload Aug 7 21:46:05.392632482 (12h6m27.136689348s) [upload=12486]: Done starting upload Aug 7 21:46:05.392784466 (12h6m27.136841327s) [upload=12486]: Uploading segments... Aug 7 21:46:05.392963140 (12h6m27.13701999s) [upload=12486]: Got next segment. Inline: false Aug 7 21:46:05.393017249 (12h6m27.137074109s) [upload=12486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:05.393023723 (12h6m27.137080582s) [upload=12486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:46:35.116242044 (12h6m56.860298909s) [upload=12486]: Next segment err: context deadline exceeded Aug 7 21:46:35.116407347 (12h6m56.860464196s) [upload=12486]: Done uploading segments... Aug 7 21:46:35.116450315 (12h6m56.860507172s) [upload=12486]: Upload finished. err: context deadline exceeded Duration: 2.683133259s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [......XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...............] duration: 902.95725ms FAIL (2025-08-07 21:54:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:41.354808211 (12h1m3.098865072s) [upload=12402]: Starting upload Aug 7 21:40:41.354938701 (12h1m3.098995568s) [upload=12402]: Done starting upload Aug 7 21:40:41.355036214 (12h1m3.099093064s) [upload=12402]: Uploading segments... Aug 7 21:40:41.355109185 (12h1m3.099166055s) [upload=12402]: Got next segment. Inline: true Aug 7 21:40:41.355116835 (12h1m3.0991737s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:40:41.355183407 (12h1m3.09924027s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:41.355190433 (12h1m3.099247297s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:41.355194357 (12h1m3.09925122s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:11.078938147 (12h1m32.822994999s) [upload=12402]: Done uploading segments... Aug 7 21:41:11.078958711 (12h1m32.823015562s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.899938086 (12h1m56.643994947s) [upload=12417]: Starting upload Aug 7 21:41:34.900014273 (12h1m56.644071133s) [upload=12417]: Done starting upload Aug 7 21:41:34.900081071 (12h1m56.644137926s) [upload=12417]: Uploading segments... Aug 7 21:41:34.900153322 (12h1m56.644210172s) [upload=12417]: Got next segment. Inline: true Aug 7 21:41:34.900158578 (12h1m56.644215438s) [upload=12417]: Waiting for error group managing segments... Aug 7 21:41:34.900228776 (12h1m56.644285623s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.900237327 (12h1m56.644294194s) [upload=12417]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.900241831 (12h1m56.644298696s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:04.627262793 (12h2m26.371319653s) [upload=12417]: Done uploading segments... Aug 7 21:42:04.627301743 (12h2m26.371358604s) [upload=12417]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:18.477725352 (12h2m40.221782248s) [upload=12428]: Starting upload Aug 7 21:42:18.477815072 (12h2m40.221871935s) [upload=12428]: Done starting upload Aug 7 21:42:18.477903321 (12h2m40.221960189s) [upload=12428]: Uploading segments... Aug 7 21:42:18.478047763 (12h2m40.222104633s) [upload=12428]: Got next segment. Inline: true Aug 7 21:42:18.478056185 (12h2m40.222113051s) [upload=12428]: Waiting for error group managing segments... Aug 7 21:42:18.478131478 (12h2m40.222188328s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:18.478138171 (12h2m40.222195021s) [upload=12428]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:18.478144895 (12h2m40.222201761s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:48.201994950 (12h3m9.946051801s) [upload=12428]: Done uploading segments... Aug 7 21:42:48.202028187 (12h3m9.946085036s) [upload=12428]: Upload finished. err: metaclient: context deadline exceeded Duration: 902.98771ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [................X.XXXXXXXXXXXXXXXXXXXXXXXX......X...........] duration: 851.754348ms FAIL (2025-08-07 21:54:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 202662 (11h58m6.212259509s) [upload=12350]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:44.877004541 (11h58m6.621061411s) [upload=12363]: Starting upload Aug 7 21:37:44.877093129 (11h58m6.621150048s) [upload=12363]: Done starting upload Aug 7 21:37:44.877179828 (11h58m6.621236677s) [upload=12363]: Uploading segments... Aug 7 21:37:44.877253719 (11h58m6.621310589s) [upload=12363]: Got next segment. Inline: true Aug 7 21:37:44.877262492 (11h58m6.621319365s) [upload=12363]: Waiting for error group managing segments... Aug 7 21:37:44.877313668 (11h58m6.621370528s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:44.877319037 (11h58m6.621375897s) [upload=12363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:44.877322512 (11h58m6.621379375s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:44.603781017 (11h59m6.347837867s) [upload=12363]: Done uploading segments... Aug 7 21:38:44.603802220 (11h59m6.347859069s) [upload=12363]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:45.032933533 (11h59m6.776990379s) [upload=12375]: Starting upload Aug 7 21:38:45.033010670 (11h59m6.777067533s) [upload=12375]: Done starting upload Aug 7 21:38:45.033084965 (11h59m6.777141811s) [upload=12375]: Uploading segments... Aug 7 21:38:45.033156998 (11h59m6.777213868s) [upload=12375]: Got next segment. Inline: true Aug 7 21:38:45.033162279 (11h59m6.777219127s) [upload=12375]: Waiting for error group managing segments... Aug 7 21:38:45.033211860 (11h59m6.777268707s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:45.033216660 (11h59m6.77727352s) [upload=12375]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:45.033219329 (11h59m6.777276175s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:44.758881739 (12h0m6.502938587s) [upload=12375]: Done uploading segments... Aug 7 21:39:44.758902501 (12h0m6.502959346s) [upload=12375]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:45.185696259 (12h0m6.929753109s) [upload=12388]: Starting upload Aug 7 21:39:45.185761209 (12h0m6.929818072s) [upload=12388]: Done starting upload Aug 7 21:39:45.185834033 (12h0m6.929890896s) [upload=12388]: Uploading segments... Aug 7 21:39:45.185982126 (12h0m6.930038997s) [upload=12388]: Got next segment. Inline: true Aug 7 21:39:45.185990859 (12h0m6.930047726s) [upload=12388]: Waiting for error group managing segments... Aug 7 21:39:45.186067966 (12h0m6.930124827s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:45.186074972 (12h0m6.930131844s) [upload=12388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:45.186078706 (12h0m6.930135567s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:44.901948337 (12h1m6.646005185s) [upload=12388]: Done uploading segments... Aug 7 21:40:44.901969954 (12h1m6.646026803s) [upload=12388]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:53.544245995 (12h1m15.288302845s) [upload=12404]: Starting upload Aug 7 21:40:53.544357406 (12h1m15.288414275s) [upload=12404]: Done starting upload Aug 7 21:40:53.544447469 (12h1m15.288504323s) [upload=12404]: Uploading segments... Aug 7 21:40:53.544512878 (12h1m15.28856975s) [upload=12404]: Got next segment. Inline: true Aug 7 21:40:53.544519444 (12h1m15.288576314s) [upload=12404]: Waiting for error group managing segments... Aug 7 21:40:53.544579423 (12h1m15.288636288s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:53.544585620 (12h1m15.288642482s) [upload=12404]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:53.544589678 (12h1m15.288646542s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:05.856628151 (12h1m27.600685011s) [upload=12404]: Done uploading segments... Aug 7 21:41:05.856646351 (12h1m27.6007032s) [upload=12404]: Upload finished. err: <nil> Aug 7 21:41:43.946896616 (12h2m5.690953463s) [upload=12420]: Starting upload Aug 7 21:41:43.946973751 (12h2m5.69103062s) [upload=12420]: Done starting upload Aug 7 21:41:43.947068595 (12h2m5.691125457s) [upload=12420]: Uploading segments... Aug 7 21:41:43.947136055 (12h2m5.691192916s) [upload=12420]: Got next segment. Inline: true Aug 7 21:41:43.947141676 (12h2m5.691198537s) [upload=12420]: Waiting for error group managing segments... Aug 7 21:41:43.947184608 (12h2m5.691241458s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:43.947190015 (12h2m5.691246865s) [upload=12420]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:43.947193244 (12h2m5.691250106s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:31.804258579 (12h2m53.548315429s) [upload=12420]: Done uploading segments... Aug 7 21:42:31.804277273 (12h2m53.548334124s) [upload=12420]: Upload finished. err: <nil> Aug 7 21:42:52.266861509 (12h3m14.01091837s) [upload=12435]: Starting upload Aug 7 21:42:52.266938245 (12h3m14.010995105s) [upload=12435]: Done starting upload Aug 7 21:42:52.267013941 (12h3m14.011070791s) [upload=12435]: Uploading segments... Aug 7 21:42:52.267054220 (12h3m14.01111109s) [upload=12435]: Got next segment. Inline: true Aug 7 21:42:52.267059450 (12h3m14.011116311s) [upload=12435]: Waiting for error group managing segments... Aug 7 21:42:52.267099382 (12h3m14.011156243s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:52.267105199 (12h3m14.011162059s) [upload=12435]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:52.267108901 (12h3m14.011165762s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:01.858360925 (12h3m23.602417792s) [upload=12435]: Done uploading segments... Aug 7 21:43:01.858412136 (12h3m23.602468987s) [upload=12435]: Upload finished. err: <nil> Aug 7 21:43:32.197096129 (12h3m53.941152979s) [upload=12448]: Starting upload Aug 7 21:43:32.197186644 (12h3m53.941243495s) [upload=12448]: Done starting upload Aug 7 21:43:32.197270859 (12h3m53.941327709s) [upload=12448]: Uploading segments... Aug 7 21:43:32.197324582 (12h3m53.941381451s) [upload=12448]: Got next segment. Inline: true Aug 7 21:43:32.197329997 (12h3m53.941386847s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:43:32.197385019 (12h3m53.941441879s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:32.197390582 (12h3m53.941447443s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:32.197394073 (12h3m53.941450934s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:32.363120575 (12h3m54.107177435s) [upload=12448]: Done uploading segments... Aug 7 21:43:32.363140640 (12h3m54.107197486s) [upload=12448]: Upload finished. err: <nil> Aug 7 21:44:28.281971344 (12h4m50.026028222s) [upload=12459]: Starting upload Aug 7 21:44:28.282137259 (12h4m50.026194124s) [upload=12459]: Done starting upload Aug 7 21:44:28.282276848 (12h4m50.026333712s) [upload=12459]: Uploading segments... Aug 7 21:44:28.282367088 (12h4m50.026423964s) [upload=12459]: Got next segment. Inline: true Aug 7 21:44:28.282375073 (12h4m50.026431948s) [upload=12459]: Waiting for error group managing segments... Aug 7 21:44:28.282442525 (12h4m50.026499391s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:28.282449059 (12h4m50.026505928s) [upload=12459]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:28.282452578 (12h4m50.026509443s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:35.643697666 (12h4m57.387754517s) [upload=12459]: Done uploading segments... Aug 7 21:44:35.643716871 (12h4m57.387773725s) [upload=12459]: Upload finished. err: <nil> Aug 7 21:45:33.044130503 (12h5m54.788187353s) [upload=12474]: Starting upload Aug 7 21:45:33.044219677 (12h5m54.788276538s) [upload=12474]: Done starting upload Aug 7 21:45:33.044292138 (12h5m54.788348988s) [upload=12474]: Uploading segments... Aug 7 21:45:33.044343424 (12h5m54.788400293s) [upload=12474]: Got next segment. Inline: true Aug 7 21:45:33.044349210 (12h5m54.78840607s) [upload=12474]: Waiting for error group managing segments... Aug 7 21:45:33.044391490 (12h5m54.78844834s) [upload=12474]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:33.044407486 (12h5m54.788464346s) [upload=12474]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:33.044413529 (12h5m54.788470389s) [upload=12474]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:33.198940222 (12h5m54.942997072s) [upload=12474]: Done uploading segments... Aug 7 21:45:33.198960825 (12h5m54.943017674s) [upload=12474]: Upload finished. err: <nil> Aug 7 21:45:57.787118330 (12h6m19.53117519s) [upload=12482]: Starting upload Aug 7 21:45:57.787191262 (12h6m19.531248122s) [upload=12482]: Done starting upload Aug 7 21:45:57.787268709 (12h6m19.53132556s) [upload=12482]: Uploading segments... Aug 7 21:45:57.787331847 (12h6m19.531388712s) [upload=12482]: Got next segment. Inline: true Aug 7 21:45:57.787339509 (12h6m19.53139637s) [upload=12482]: Waiting for error group managing segments... Aug 7 21:45:57.787386486 (12h6m19.531443346s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:57.787403824 (12h6m19.531460685s) [upload=12482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:57.787407765 (12h6m19.531464626s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:46:57.502016096 (12h7m19.246072946s) [upload=12482]: Done uploading segments... Aug 7 21:46:57.502068445 (12h7m19.246125296s) [upload=12482]: Upload finished. err: metaclient: context deadline exceeded Duration: 851.782522ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 40 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X.XX............] duration: 906.367849ms FAIL (2025-08-07 21:54:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 906.40512ms core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [....................X............X.X.X......................] duration: 105.070719ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:54:12 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 105.104839ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................] duration: 1.197442192s FAIL (2025-08-07 21:54:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.197468741s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [..X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................] duration: 2.783462159s FAIL (2025-08-07 21:54:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.783490726s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................XXXX........] duration: 3.239899287s FAIL (2025-08-07 21:54:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== tchRequestItem_ObjectBegin Aug 7 21:24:45.813454815 (11h45m7.557511676s) [upload=12171]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:43.107142097 (11h46m4.851198968s) [seg_pos={0 0}, upload=12171]: Begin upload segment... Aug 7 21:25:43.107205391 (11h46m4.851262241s) [seg_pos={0 0}, upload=12171]: Done begin upload segment. Aug 7 21:25:43.107237087 (11h46m4.851293957s) [noise=true , address=104.36.82.15:4269 , piece_id=HTXIUN5HOU4B66MPMEXJJSUL6JNBE6FI6EB62IEG7P3WTZLAH47Q, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.107267905 (11h46m4.851324768s) [noise=true , address=104.36.82.8:4983 , piece_id=ZDR4NMXM2MJTWTLT5776GQIUXCIY7TMRNBQ43L73XOFFBLSA6KQQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.107830175 (11h46m4.851887046s) [noise=true , address=45.91.116.7:4013 , piece_id=JEWYLXASDWIZ6G3LSHWMUAFAZIKLWYO7KOZLEQEGNZB3HMNVSKYA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.108780038 (11h46m4.852836904s) [noise=true , address=104.36.82.16:4319 , piece_id=36CAJ2JJZF3CLN67PLA7VUM2F6NULTFSQ3WG6EDE734G3LNFZJTA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.108959605 (11h46m4.853016472s) [noise=true , address=104.36.82.43:4852 , piece_id=NBGNTQUWEOVY72D3XFFO74EFUL52BZP222Z6FNIYCFB43FOBIW2Q, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.109220359 (11h46m4.853277218s) [noise=true , address=79.127.201.214:4296 , piece_id=SK5JKFXZVF5XRW6STC5KDH5WNI7JHB6X5X5MLZ6WLLHG5SO6APSA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.109377214 (11h46m4.853434079s) [noise=true , address=104.36.82.37:4662 , piece_id=KZG6JYEDQUTQGQCTUAZK7R3JRD7GKFLKORVEC5LIZ47GVJ6MFZRA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.109626289 (11h46m4.853683159s) [noise=true , address=104.36.82.13:4195 , piece_id=YVPFOIDCG5F4LET7V5SZU2X6WKD4MCZWIQ6HUE7NVIOIVBRGDNHA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.109743295 (11h46m4.853800166s) [noise=true , address=79.127.205.245:4234 , piece_id=RADEPAKA2ZEQSI5UUX5BHRAR5JHNMOQI7MGZW42A4ZW52KJOXBRA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.110014430 (11h46m4.854071298s) [noise=true , address=64.49.13.138:4487 , piece_id=MXQL4B6PL3FOQS4WCCGCPF5DT5C3JDLYJNOEIE7GKNE5FRQJZ7EA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.110156657 (11h46m4.854213527s) [noise=true , address=45.91.116.8:4056 , piece_id=DOFJMEXQQBOTVMCWC77YLLDCUY4XYYRM3X5BOBSXXAPB7T4YVHZQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.110413863 (11h46m4.854470734s) [noise=true , address=104.36.82.35:4607 , piece_id=DQKF7V3EW6OT5C4IFITEQNOXUUHDOMG5AESWXUYJP3SERMWUF5VQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.110778464 (11h46m4.854835327s) [noise=true , address=104.36.82.38:4708 , piece_id=JUSE5AFFHKPHCDVDRN5WWWXQK6VHIY3CXQWMJ4EXEJXVMICMZIXA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.111117136 (11h46m4.855173999s) [noise=true , address=45.91.116.14:4197 , piece_id=52LI7AMEDE2OBW6TAIOP5Y2M4J4HAWEITEKIYPZZ7XQ2Z5CGYIBQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.111560812 (11h46m4.855617682s) [noise=true , address=104.36.82.35:4602 , piece_id=42ZEPU7YI52FBKWCUG2ZRDLPOLG5IYXSPR4HJ2QUSURO65WIAICQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.111976240 (11h46m4.856033107s) [noise=true , address=104.36.82.51:4073 , piece_id=HPC5WAIPHCOMTZDKPDPGZ7UJ43KPZZ2IFL5VFH7ILXHCF63D5NXA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.112141257 (11h46m4.856198127s) [noise=true , address=104.36.82.22:4511 , piece_id=WWW6R3JFLEQCGIAIZU6VXOC6XCPXFC3AWJ74PBL4PQXKTTSF535A, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.112560277 (11h46m4.856617148s) [noise=true , address=104.36.82.62:4412 , piece_id=6NAJYYYU3AX7KJNFBXTPEYRCFFTMV3ESLVFUWTEGKGNOEHFHJJKA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.675139110 (11h46m5.419195972s) [upload=12171]: Next returned nil segment Aug 7 21:25:43.675161049 (11h46m5.419217914s) [upload=12171]: Waiting for error group managing segments... Aug 7 21:25:43.872519019 (11h46m5.616575886s) [noise=true , address=104.36.82.62:4412 , piece_id=6NAJYYYU3AX7KJNFBXTPEYRCFFTMV3ESLVFUWTEGKGNOEHFHJJKA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.876126423 (11h46m5.62018327s) [noise=true , address=104.36.82.51:4073 , piece_id=HPC5WAIPHCOMTZDKPDPGZ7UJ43KPZZ2IFL5VFH7ILXHCF63D5NXA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.882439205 (11h46m5.626496076s) [noise=true , address=104.36.82.35:4602 , piece_id=42ZEPU7YI52FBKWCUG2ZRDLPOLG5IYXSPR4HJ2QUSURO65WIAICQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.886290836 (11h46m5.63034769s) [noise=true , address=104.36.82.22:4511 , piece_id=WWW6R3JFLEQCGIAIZU6VXOC6XCPXFC3AWJ74PBL4PQXKTTSF535A, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.888219892 (11h46m5.632276738s) [noise=true , address=104.36.82.15:4269 , piece_id=HTXIUN5HOU4B66MPMEXJJSUL6JNBE6FI6EB62IEG7P3WTZLAH47Q, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.888421317 (11h46m5.632478185s) [noise=true , address=45.91.116.14:4197 , piece_id=52LI7AMEDE2OBW6TAIOP5Y2M4J4HAWEITEKIYPZZ7XQ2Z5CGYIBQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.889366258 (11h46m5.633423104s) [noise=true , address=104.36.82.35:4607 , piece_id=DQKF7V3EW6OT5C4IFITEQNOXUUHDOMG5AESWXUYJP3SERMWUF5VQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.890288566 (11h46m5.634345437s) [noise=true , address=104.36.82.43:4852 , piece_id=NBGNTQUWEOVY72D3XFFO74EFUL52BZP222Z6FNIYCFB43FOBIW2Q, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.893016005 (11h46m5.637072875s) [noise=true , address=45.91.116.8:4056 , piece_id=DOFJMEXQQBOTVMCWC77YLLDCUY4XYYRM3X5BOBSXXAPB7T4YVHZQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.903527905 (11h46m5.647584771s) [noise=true , address=104.36.82.38:4708 , piece_id=JUSE5AFFHKPHCDVDRN5WWWXQK6VHIY3CXQWMJ4EXEJXVMICMZIXA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.912802006 (11h46m5.656858856s) [noise=true , address=45.91.116.7:4013 , piece_id=JEWYLXASDWIZ6G3LSHWMUAFAZIKLWYO7KOZLEQEGNZB3HMNVSKYA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.974200245 (11h46m5.718257092s) [noise=true , address=104.36.82.8:4983 , piece_id=ZDR4NMXM2MJTWTLT5776GQIUXCIY7TMRNBQ43L73XOFFBLSA6KQQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:44.128893332 (11h46m5.872950195s) [noise=true , address=79.127.201.214:4296 , piece_id=SK5JKFXZVF5XRW6STC5KDH5WNI7JHB6X5X5MLZ6WLLHG5SO6APSA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:44.143902251 (11h46m5.887959098s) [noise=true , address=104.36.82.37:4662 , piece_id=KZG6JYEDQUTQGQCTUAZK7R3JRD7GKFLKORVEC5LIZ47GVJ6MFZRA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:44.143991645 (11h46m5.888048495s) [seg_pos={0 0}, upload=12171]: Segment reached optimal threshold of 14 pieces. Aug 7 21:25:44.144122598 (11h46m5.888179458s) [noise=true , address=64.49.13.138:4487 , piece_id=MXQL4B6PL3FOQS4WCCGCPF5DT5C3JDLYJNOEIE7GKNE5FRQJZ7EA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF Aug 7 21:25:44.144194345 (11h46m5.888251205s) [noise=true , address=79.127.205.245:4234 , piece_id=RADEPAKA2ZEQSI5UUX5BHRAR5JHNMOQI7MGZW42A4ZW52KJOXBRA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJWq44d6y4Ai4tyZpCUvCgZeMHrVdiiSSngTR75YZZXbsgq2d): protocol: expected piece hash; context canceled; EOF Aug 7 21:25:44.144213145 (11h46m5.888269992s) [noise=true , address=104.36.82.13:4195 , piece_id=YVPFOIDCG5F4LET7V5SZU2X6WKD4MCZWIQ6HUE7NVIOIVBRGDNHA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH): protocol: write tcp 139.59.144.22:41808->104.36.82.13:4195: use of closed network connection; write tcp 139.59.144.22:41808->104.36.82.13:4195: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41808->104.36.82.13:4195: use of closed network connection Aug 7 21:25:44.144301767 (11h46m5.888358613s) [noise=true , address=104.36.82.16:4319 , piece_id=36CAJ2JJZF3CLN67PLA7VUM2F6NULTFSQ3WG6EDE734G3LNFZJTA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hQTG2DzecaoN1Na2mY3JJqNj2qxtJDQkhMYmRmd9hve8D7T9C): protocol: write tcp 139.59.144.22:51894->104.36.82.16:4319: use of closed network connection; write tcp 139.59.144.22:51894->104.36.82.16:4319: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51894->104.36.82.16:4319: use of closed network connection Aug 7 21:25:44.144323393 (11h46m5.88838024s) [seg_pos={0 0}, upload=12171]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:25:44.144394152 (11h46m5.888450998s) [upload=12171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:25:44.144398730 (11h46m5.888455591s) [upload=12171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:47.207768845 (12h9m8.951825696s) [upload=12171]: Done uploading segments... Aug 7 21:48:47.207799297 (12h9m8.951856149s) [upload=12171]: Upload finished. err: metaclient: context deadline exceeded Duration:* 3.239928776s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXX....X............] duration: 2.558330448s FAIL (2025-08-07 21:54:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== :35.590125129 (12h5m57.33418199s) [upload=12475]: Got next segment. Inline: false Aug 7 21:45:35.590176001 (12h5m57.33423285s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:35.590181518 (12h5m57.334238369s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:35.699013188 (12h5m57.443070052s) [seg_pos={0 0}, upload=12475]: Begin upload segment... Aug 7 21:45:35.699066084 (12h5m57.443122935s) [seg_pos={0 0}, upload=12475]: Done begin upload segment. Aug 7 21:45:35.699091982 (12h5m57.443148842s) [noise=true , address=77.90.31.17:4771 , piece_id=LDTJCTYFHCPL2SHCYONX3SJWBX3GME4GGILSOCHZWG67HKEAHDYQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.699635648 (12h5m57.443692518s) [noise=true , address=104.36.82.11:4115 , piece_id=U6HOQABV6JV5T4MFFT3EDQF6R77LXTPWZ33FJWUEYOWLWLPFLYWA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700048642 (12h5m57.444105509s) [noise=true , address=104.36.82.5:4840 , piece_id=V3VLVQPAAHDACW2WGUPYAGVDU2NRGR4FLQD2SFKVP2XLQALKP3SA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700358698 (12h5m57.444415561s) [noise=true , address=45.91.116.5:4299 , piece_id=YK7OVMQRE2PYUS2LDZOZGBKYV42U6VDWI2D6CMI4D2CIEV2DBACQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700672646 (12h5m57.444729509s) [noise=true , address=45.91.116.14:4191 , piece_id=TYBU2DWWTVWWEEMTBD2PIQKVIU3YRHKXH7ACD7Y3VKZVDXPDRALA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701028296 (12h5m57.445085159s) [noise=true , address=104.36.82.13:4187 , piece_id=WCIN3NV6AFBYDUGO7TDPOMZG5YJ24ISXVB2WSLB46EZOPPGQKW2Q, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701400580 (12h5m57.44545744s) [noise=true , address=77.90.31.15:4697 , piece_id=XQ5CZFWWVUZ6MKSIYZVOUZG6YAA7UAVVISSKVAPJXXD7HPXJHK3A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701701900 (12h5m57.445758763s) [noise=true , address=45.91.116.11:4119 , piece_id=K5FKXMYXUWHRKFZPDHFKZ425RAZLEQH254K65QOITKTC4APUYL7A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.702301822 (12h5m57.446358688s) [noise=true , address=45.91.116.11:4118 , piece_id=QPX3R57IYGZFNNOCPHOSQCRRQH6KJMDYSBPGHLX34DNABYYUXZ6A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.702831318 (12h5m57.446888179s) [noise=true , address=104.36.82.10:4078 , piece_id=BEMP3VUCCLIJAY2OU2TFDJ5SQDJFO4OGL3UEHQ7YPMJGBRZ4E6BQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.703272004 (12h5m57.447328915s) [noise=true , address=45.91.116.6:4974 , piece_id=5HPCLRLKSQWUBMWQXW5LEOVQVBIKV62QAYF4YKQGQREWT3ZTAI2Q, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.703720399 (12h5m57.447777259s) [noise=true , address=104.36.82.3:4767 , piece_id=D3DBXBQ3UNQCWF5DR3HV46XZSOMRQHR33SQ6VSA7BXY7OZEWXNAA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704065340 (12h5m57.44812221s) [noise=true , address=77.90.31.13:4626 , piece_id=KM7OLI4GIFFIK4UTVRX26KMGVRZMASK47B7B7FOYCS6DU5SMZEXA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704350352 (12h5m57.448407212s) [noise=true , address=104.36.82.12:4151 , piece_id=UUTIELJOX7CUEFHHFWDD4XETR3EB7IQOCKUZXQALKKGXPBVWH4TA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704707337 (12h5m57.448764198s) [noise=true , address=77.90.31.12:4590 , piece_id=BF63RJ2G676UQFZGWIAXFCZR2DC67F4AH5VZAB3MYIDDOMJKGISA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.705111011 (12h5m57.449167878s) [noise=true , address=104.36.82.7:4911 , piece_id=6HSMRZBG26Y2XZ7NQZ6JD7P2BXZTDFDC6A5JIIS7SM7T6FAL2PWQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:36.332909886 (12h5m58.076966752s) [upload=12475]: Next returned nil segment Aug 7 21:45:36.332933437 (12h5m58.076990311s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:45:36.489772849 (12h5m58.233829715s) [noise=true , address=104.36.82.10:4078 , piece_id=BEMP3VUCCLIJAY2OU2TFDJ5SQDJFO4OGL3UEHQ7YPMJGBRZ4E6BQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.509052197 (12h5m58.253109052s) [noise=true , address=104.36.82.12:4151 , piece_id=UUTIELJOX7CUEFHHFWDD4XETR3EB7IQOCKUZXQALKKGXPBVWH4TA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.542228341 (12h5m58.286285191s) [noise=true , address=77.90.31.12:4590 , piece_id=BF63RJ2G676UQFZGWIAXFCZR2DC67F4AH5VZAB3MYIDDOMJKGISA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.573060953 (12h5m58.317117805s) [noise=true , address=45.91.116.6:4974 , piece_id=5HPCLRLKSQWUBMWQXW5LEOVQVBIKV62QAYF4YKQGQREWT3ZTAI2Q, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.573821057 (12h5m58.317877923s) [noise=true , address=45.91.116.5:4299 , piece_id=YK7OVMQRE2PYUS2LDZOZGBKYV42U6VDWI2D6CMI4D2CIEV2DBACQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.620961481 (12h5m58.365018331s) [noise=true , address=77.90.31.17:4771 , piece_id=LDTJCTYFHCPL2SHCYONX3SJWBX3GME4GGILSOCHZWG67HKEAHDYQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.631043610 (12h5m58.375100483s) [noise=true , address=45.91.116.14:4191 , piece_id=TYBU2DWWTVWWEEMTBD2PIQKVIU3YRHKXH7ACD7Y3VKZVDXPDRALA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.632954530 (12h5m58.377011377s) [noise=true , address=45.91.116.11:4119 , piece_id=K5FKXMYXUWHRKFZPDHFKZ425RAZLEQH254K65QOITKTC4APUYL7A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.644557641 (12h5m58.388614491s) [noise=true , address=104.36.82.7:4911 , piece_id=6HSMRZBG26Y2XZ7NQZ6JD7P2BXZTDFDC6A5JIIS7SM7T6FAL2PWQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.649884441 (12h5m58.393941291s) [noise=true , address=77.90.31.13:4626 , piece_id=KM7OLI4GIFFIK4UTVRX26KMGVRZMASK47B7B7FOYCS6DU5SMZEXA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.661770778 (12h5m58.405827645s) [noise=true , address=77.90.31.15:4697 , piece_id=XQ5CZFWWVUZ6MKSIYZVOUZG6YAA7UAVVISSKVAPJXXD7HPXJHK3A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.725730770 (12h5m58.46978762s) [noise=true , address=104.36.82.3:4767 , piece_id=D3DBXBQ3UNQCWF5DR3HV46XZSOMRQHR33SQ6VSA7BXY7OZEWXNAA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.869176760 (12h5m58.613233628s) [noise=true , address=45.91.116.11:4118 , piece_id=QPX3R57IYGZFNNOCPHOSQCRRQH6KJMDYSBPGHLX34DNABYYUXZ6A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.869206102 (12h5m58.61326296s) [seg_pos={0 0}, upload=12475]: Segment reached optimal threshold of 13 pieces. Aug 7 21:45:36.869561423 (12h5m58.613618286s) [noise=true , address=104.36.82.11:4115 , piece_id=U6HOQABV6JV5T4MFFT3EDQF6R77LXTPWZ33FJWUEYOWLWLPFLYWA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Aug 7 21:45:36.869806552 (12h5m58.613863414s) [noise=true , address=104.36.82.5:4840 , piece_id=V3VLVQPAAHDACW2WGUPYAGVDU2NRGR4FLQD2SFKVP2XLQALKP3SA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection; write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection Aug 7 21:45:36.869995683 (12h5m58.614052547s) [noise=true , address=104.36.82.13:4187 , piece_id=WCIN3NV6AFBYDUGO7TDPOMZG5YJ24ISXVB2WSLB46EZOPPGQKW2Q, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection Aug 7 21:45:36.870032790 (12h5m58.614089658s) [seg_pos={0 0}, upload=12475]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:45:36.870148681 (12h5m58.614205541s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:45:36.870154041 (12h5m58.614210892s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:37.015978570 (12h5m58.76003542s) [upload=12475]: Done uploading segments... Aug 7 21:45:37.016019019 (12h5m58.760075869s) [upload=12475]: Upload finished. err: <nil> Aug 7 21:46:05.392535892 (12h6m27.136592742s) [upload=12486]: Starting upload Aug 7 21:46:05.392632482 (12h6m27.136689348s) [upload=12486]: Done starting upload Aug 7 21:46:05.392784466 (12h6m27.136841327s) [upload=12486]: Uploading segments... Aug 7 21:46:05.392963140 (12h6m27.13701999s) [upload=12486]: Got next segment. Inline: false Aug 7 21:46:05.393017249 (12h6m27.137074109s) [upload=12486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:05.393023723 (12h6m27.137080582s) [upload=12486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:46:35.116242044 (12h6m56.860298909s) [upload=12486]: Next segment err: context deadline exceeded Aug 7 21:46:35.116407347 (12h6m56.860464196s) [upload=12486]: Done uploading segments... Aug 7 21:46:35.116450315 (12h6m56.860507172s) [upload=12486]: Upload finished. err: context deadline exceeded Duration: 2.558402199s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXX......X.X.........] duration: 2.825305285s FAIL (2025-08-07 21:54:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.825339812s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.....................X............X.X.X.....................] duration: 105.070464ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:53:51 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 105.096003ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [...X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...............] duration: 2.50377119s FAIL (2025-08-07 21:53:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.503797781s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXX....X...........] duration: 2.712919658s FAIL (2025-08-07 21:53:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== :35.590125129 (12h5m57.33418199s) [upload=12475]: Got next segment. Inline: false Aug 7 21:45:35.590176001 (12h5m57.33423285s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:35.590181518 (12h5m57.334238369s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:35.699013188 (12h5m57.443070052s) [seg_pos={0 0}, upload=12475]: Begin upload segment... Aug 7 21:45:35.699066084 (12h5m57.443122935s) [seg_pos={0 0}, upload=12475]: Done begin upload segment. Aug 7 21:45:35.699091982 (12h5m57.443148842s) [noise=true , address=77.90.31.17:4771 , piece_id=LDTJCTYFHCPL2SHCYONX3SJWBX3GME4GGILSOCHZWG67HKEAHDYQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.699635648 (12h5m57.443692518s) [noise=true , address=104.36.82.11:4115 , piece_id=U6HOQABV6JV5T4MFFT3EDQF6R77LXTPWZ33FJWUEYOWLWLPFLYWA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700048642 (12h5m57.444105509s) [noise=true , address=104.36.82.5:4840 , piece_id=V3VLVQPAAHDACW2WGUPYAGVDU2NRGR4FLQD2SFKVP2XLQALKP3SA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700358698 (12h5m57.444415561s) [noise=true , address=45.91.116.5:4299 , piece_id=YK7OVMQRE2PYUS2LDZOZGBKYV42U6VDWI2D6CMI4D2CIEV2DBACQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700672646 (12h5m57.444729509s) [noise=true , address=45.91.116.14:4191 , piece_id=TYBU2DWWTVWWEEMTBD2PIQKVIU3YRHKXH7ACD7Y3VKZVDXPDRALA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701028296 (12h5m57.445085159s) [noise=true , address=104.36.82.13:4187 , piece_id=WCIN3NV6AFBYDUGO7TDPOMZG5YJ24ISXVB2WSLB46EZOPPGQKW2Q, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701400580 (12h5m57.44545744s) [noise=true , address=77.90.31.15:4697 , piece_id=XQ5CZFWWVUZ6MKSIYZVOUZG6YAA7UAVVISSKVAPJXXD7HPXJHK3A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701701900 (12h5m57.445758763s) [noise=true , address=45.91.116.11:4119 , piece_id=K5FKXMYXUWHRKFZPDHFKZ425RAZLEQH254K65QOITKTC4APUYL7A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.702301822 (12h5m57.446358688s) [noise=true , address=45.91.116.11:4118 , piece_id=QPX3R57IYGZFNNOCPHOSQCRRQH6KJMDYSBPGHLX34DNABYYUXZ6A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.702831318 (12h5m57.446888179s) [noise=true , address=104.36.82.10:4078 , piece_id=BEMP3VUCCLIJAY2OU2TFDJ5SQDJFO4OGL3UEHQ7YPMJGBRZ4E6BQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.703272004 (12h5m57.447328915s) [noise=true , address=45.91.116.6:4974 , piece_id=5HPCLRLKSQWUBMWQXW5LEOVQVBIKV62QAYF4YKQGQREWT3ZTAI2Q, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.703720399 (12h5m57.447777259s) [noise=true , address=104.36.82.3:4767 , piece_id=D3DBXBQ3UNQCWF5DR3HV46XZSOMRQHR33SQ6VSA7BXY7OZEWXNAA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704065340 (12h5m57.44812221s) [noise=true , address=77.90.31.13:4626 , piece_id=KM7OLI4GIFFIK4UTVRX26KMGVRZMASK47B7B7FOYCS6DU5SMZEXA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704350352 (12h5m57.448407212s) [noise=true , address=104.36.82.12:4151 , piece_id=UUTIELJOX7CUEFHHFWDD4XETR3EB7IQOCKUZXQALKKGXPBVWH4TA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704707337 (12h5m57.448764198s) [noise=true , address=77.90.31.12:4590 , piece_id=BF63RJ2G676UQFZGWIAXFCZR2DC67F4AH5VZAB3MYIDDOMJKGISA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.705111011 (12h5m57.449167878s) [noise=true , address=104.36.82.7:4911 , piece_id=6HSMRZBG26Y2XZ7NQZ6JD7P2BXZTDFDC6A5JIIS7SM7T6FAL2PWQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:36.332909886 (12h5m58.076966752s) [upload=12475]: Next returned nil segment Aug 7 21:45:36.332933437 (12h5m58.076990311s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:45:36.489772849 (12h5m58.233829715s) [noise=true , address=104.36.82.10:4078 , piece_id=BEMP3VUCCLIJAY2OU2TFDJ5SQDJFO4OGL3UEHQ7YPMJGBRZ4E6BQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.509052197 (12h5m58.253109052s) [noise=true , address=104.36.82.12:4151 , piece_id=UUTIELJOX7CUEFHHFWDD4XETR3EB7IQOCKUZXQALKKGXPBVWH4TA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.542228341 (12h5m58.286285191s) [noise=true , address=77.90.31.12:4590 , piece_id=BF63RJ2G676UQFZGWIAXFCZR2DC67F4AH5VZAB3MYIDDOMJKGISA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.573060953 (12h5m58.317117805s) [noise=true , address=45.91.116.6:4974 , piece_id=5HPCLRLKSQWUBMWQXW5LEOVQVBIKV62QAYF4YKQGQREWT3ZTAI2Q, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.573821057 (12h5m58.317877923s) [noise=true , address=45.91.116.5:4299 , piece_id=YK7OVMQRE2PYUS2LDZOZGBKYV42U6VDWI2D6CMI4D2CIEV2DBACQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.620961481 (12h5m58.365018331s) [noise=true , address=77.90.31.17:4771 , piece_id=LDTJCTYFHCPL2SHCYONX3SJWBX3GME4GGILSOCHZWG67HKEAHDYQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.631043610 (12h5m58.375100483s) [noise=true , address=45.91.116.14:4191 , piece_id=TYBU2DWWTVWWEEMTBD2PIQKVIU3YRHKXH7ACD7Y3VKZVDXPDRALA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.632954530 (12h5m58.377011377s) [noise=true , address=45.91.116.11:4119 , piece_id=K5FKXMYXUWHRKFZPDHFKZ425RAZLEQH254K65QOITKTC4APUYL7A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.644557641 (12h5m58.388614491s) [noise=true , address=104.36.82.7:4911 , piece_id=6HSMRZBG26Y2XZ7NQZ6JD7P2BXZTDFDC6A5JIIS7SM7T6FAL2PWQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.649884441 (12h5m58.393941291s) [noise=true , address=77.90.31.13:4626 , piece_id=KM7OLI4GIFFIK4UTVRX26KMGVRZMASK47B7B7FOYCS6DU5SMZEXA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.661770778 (12h5m58.405827645s) [noise=true , address=77.90.31.15:4697 , piece_id=XQ5CZFWWVUZ6MKSIYZVOUZG6YAA7UAVVISSKVAPJXXD7HPXJHK3A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.725730770 (12h5m58.46978762s) [noise=true , address=104.36.82.3:4767 , piece_id=D3DBXBQ3UNQCWF5DR3HV46XZSOMRQHR33SQ6VSA7BXY7OZEWXNAA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.869176760 (12h5m58.613233628s) [noise=true , address=45.91.116.11:4118 , piece_id=QPX3R57IYGZFNNOCPHOSQCRRQH6KJMDYSBPGHLX34DNABYYUXZ6A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.869206102 (12h5m58.61326296s) [seg_pos={0 0}, upload=12475]: Segment reached optimal threshold of 13 pieces. Aug 7 21:45:36.869561423 (12h5m58.613618286s) [noise=true , address=104.36.82.11:4115 , piece_id=U6HOQABV6JV5T4MFFT3EDQF6R77LXTPWZ33FJWUEYOWLWLPFLYWA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Aug 7 21:45:36.869806552 (12h5m58.613863414s) [noise=true , address=104.36.82.5:4840 , piece_id=V3VLVQPAAHDACW2WGUPYAGVDU2NRGR4FLQD2SFKVP2XLQALKP3SA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection; write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection Aug 7 21:45:36.869995683 (12h5m58.614052547s) [noise=true , address=104.36.82.13:4187 , piece_id=WCIN3NV6AFBYDUGO7TDPOMZG5YJ24ISXVB2WSLB46EZOPPGQKW2Q, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection Aug 7 21:45:36.870032790 (12h5m58.614089658s) [seg_pos={0 0}, upload=12475]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:45:36.870148681 (12h5m58.614205541s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:45:36.870154041 (12h5m58.614210892s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:37.015978570 (12h5m58.76003542s) [upload=12475]: Done uploading segments... Aug 7 21:45:37.016019019 (12h5m58.760075869s) [upload=12475]: Upload finished. err: <nil> Aug 7 21:46:05.392535892 (12h6m27.136592742s) [upload=12486]: Starting upload Aug 7 21:46:05.392632482 (12h6m27.136689348s) [upload=12486]: Done starting upload Aug 7 21:46:05.392784466 (12h6m27.136841327s) [upload=12486]: Uploading segments... Aug 7 21:46:05.392963140 (12h6m27.13701999s) [upload=12486]: Got next segment. Inline: false Aug 7 21:46:05.393017249 (12h6m27.137074109s) [upload=12486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:05.393023723 (12h6m27.137080582s) [upload=12486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:46:35.116242044 (12h6m56.860298909s) [upload=12486]: Next segment err: context deadline exceeded Aug 7 21:46:35.116407347 (12h6m56.860464196s) [upload=12486]: Done uploading segments... Aug 7 21:46:35.116450315 (12h6m56.860507172s) [upload=12486]: Upload finished. err: context deadline exceeded Duration: 2.712945744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [.................X.XXXXXXXXXXXXXXXXXXXXXXXX......X..........] duration: 1.084378096s FAIL (2025-08-07 21:53:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 202662 (11h58m6.212259509s) [upload=12350]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:44.877004541 (11h58m6.621061411s) [upload=12363]: Starting upload Aug 7 21:37:44.877093129 (11h58m6.621150048s) [upload=12363]: Done starting upload Aug 7 21:37:44.877179828 (11h58m6.621236677s) [upload=12363]: Uploading segments... Aug 7 21:37:44.877253719 (11h58m6.621310589s) [upload=12363]: Got next segment. Inline: true Aug 7 21:37:44.877262492 (11h58m6.621319365s) [upload=12363]: Waiting for error group managing segments... Aug 7 21:37:44.877313668 (11h58m6.621370528s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:44.877319037 (11h58m6.621375897s) [upload=12363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:44.877322512 (11h58m6.621379375s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:44.603781017 (11h59m6.347837867s) [upload=12363]: Done uploading segments... Aug 7 21:38:44.603802220 (11h59m6.347859069s) [upload=12363]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:45.032933533 (11h59m6.776990379s) [upload=12375]: Starting upload Aug 7 21:38:45.033010670 (11h59m6.777067533s) [upload=12375]: Done starting upload Aug 7 21:38:45.033084965 (11h59m6.777141811s) [upload=12375]: Uploading segments... Aug 7 21:38:45.033156998 (11h59m6.777213868s) [upload=12375]: Got next segment. Inline: true Aug 7 21:38:45.033162279 (11h59m6.777219127s) [upload=12375]: Waiting for error group managing segments... Aug 7 21:38:45.033211860 (11h59m6.777268707s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:45.033216660 (11h59m6.77727352s) [upload=12375]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:45.033219329 (11h59m6.777276175s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:44.758881739 (12h0m6.502938587s) [upload=12375]: Done uploading segments... Aug 7 21:39:44.758902501 (12h0m6.502959346s) [upload=12375]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:45.185696259 (12h0m6.929753109s) [upload=12388]: Starting upload Aug 7 21:39:45.185761209 (12h0m6.929818072s) [upload=12388]: Done starting upload Aug 7 21:39:45.185834033 (12h0m6.929890896s) [upload=12388]: Uploading segments... Aug 7 21:39:45.185982126 (12h0m6.930038997s) [upload=12388]: Got next segment. Inline: true Aug 7 21:39:45.185990859 (12h0m6.930047726s) [upload=12388]: Waiting for error group managing segments... Aug 7 21:39:45.186067966 (12h0m6.930124827s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:45.186074972 (12h0m6.930131844s) [upload=12388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:45.186078706 (12h0m6.930135567s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:44.901948337 (12h1m6.646005185s) [upload=12388]: Done uploading segments... Aug 7 21:40:44.901969954 (12h1m6.646026803s) [upload=12388]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:53.544245995 (12h1m15.288302845s) [upload=12404]: Starting upload Aug 7 21:40:53.544357406 (12h1m15.288414275s) [upload=12404]: Done starting upload Aug 7 21:40:53.544447469 (12h1m15.288504323s) [upload=12404]: Uploading segments... Aug 7 21:40:53.544512878 (12h1m15.28856975s) [upload=12404]: Got next segment. Inline: true Aug 7 21:40:53.544519444 (12h1m15.288576314s) [upload=12404]: Waiting for error group managing segments... Aug 7 21:40:53.544579423 (12h1m15.288636288s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:53.544585620 (12h1m15.288642482s) [upload=12404]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:53.544589678 (12h1m15.288646542s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:05.856628151 (12h1m27.600685011s) [upload=12404]: Done uploading segments... Aug 7 21:41:05.856646351 (12h1m27.6007032s) [upload=12404]: Upload finished. err: <nil> Aug 7 21:41:43.946896616 (12h2m5.690953463s) [upload=12420]: Starting upload Aug 7 21:41:43.946973751 (12h2m5.69103062s) [upload=12420]: Done starting upload Aug 7 21:41:43.947068595 (12h2m5.691125457s) [upload=12420]: Uploading segments... Aug 7 21:41:43.947136055 (12h2m5.691192916s) [upload=12420]: Got next segment. Inline: true Aug 7 21:41:43.947141676 (12h2m5.691198537s) [upload=12420]: Waiting for error group managing segments... Aug 7 21:41:43.947184608 (12h2m5.691241458s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:43.947190015 (12h2m5.691246865s) [upload=12420]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:43.947193244 (12h2m5.691250106s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:31.804258579 (12h2m53.548315429s) [upload=12420]: Done uploading segments... Aug 7 21:42:31.804277273 (12h2m53.548334124s) [upload=12420]: Upload finished. err: <nil> Aug 7 21:42:52.266861509 (12h3m14.01091837s) [upload=12435]: Starting upload Aug 7 21:42:52.266938245 (12h3m14.010995105s) [upload=12435]: Done starting upload Aug 7 21:42:52.267013941 (12h3m14.011070791s) [upload=12435]: Uploading segments... Aug 7 21:42:52.267054220 (12h3m14.01111109s) [upload=12435]: Got next segment. Inline: true Aug 7 21:42:52.267059450 (12h3m14.011116311s) [upload=12435]: Waiting for error group managing segments... Aug 7 21:42:52.267099382 (12h3m14.011156243s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:52.267105199 (12h3m14.011162059s) [upload=12435]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:52.267108901 (12h3m14.011165762s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:01.858360925 (12h3m23.602417792s) [upload=12435]: Done uploading segments... Aug 7 21:43:01.858412136 (12h3m23.602468987s) [upload=12435]: Upload finished. err: <nil> Aug 7 21:43:32.197096129 (12h3m53.941152979s) [upload=12448]: Starting upload Aug 7 21:43:32.197186644 (12h3m53.941243495s) [upload=12448]: Done starting upload Aug 7 21:43:32.197270859 (12h3m53.941327709s) [upload=12448]: Uploading segments... Aug 7 21:43:32.197324582 (12h3m53.941381451s) [upload=12448]: Got next segment. Inline: true Aug 7 21:43:32.197329997 (12h3m53.941386847s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:43:32.197385019 (12h3m53.941441879s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:32.197390582 (12h3m53.941447443s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:32.197394073 (12h3m53.941450934s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:32.363120575 (12h3m54.107177435s) [upload=12448]: Done uploading segments... Aug 7 21:43:32.363140640 (12h3m54.107197486s) [upload=12448]: Upload finished. err: <nil> Aug 7 21:44:28.281971344 (12h4m50.026028222s) [upload=12459]: Starting upload Aug 7 21:44:28.282137259 (12h4m50.026194124s) [upload=12459]: Done starting upload Aug 7 21:44:28.282276848 (12h4m50.026333712s) [upload=12459]: Uploading segments... Aug 7 21:44:28.282367088 (12h4m50.026423964s) [upload=12459]: Got next segment. Inline: true Aug 7 21:44:28.282375073 (12h4m50.026431948s) [upload=12459]: Waiting for error group managing segments... Aug 7 21:44:28.282442525 (12h4m50.026499391s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:28.282449059 (12h4m50.026505928s) [upload=12459]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:28.282452578 (12h4m50.026509443s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:35.643697666 (12h4m57.387754517s) [upload=12459]: Done uploading segments... Aug 7 21:44:35.643716871 (12h4m57.387773725s) [upload=12459]: Upload finished. err: <nil> Aug 7 21:45:33.044130503 (12h5m54.788187353s) [upload=12474]: Starting upload Aug 7 21:45:33.044219677 (12h5m54.788276538s) [upload=12474]: Done starting upload Aug 7 21:45:33.044292138 (12h5m54.788348988s) [upload=12474]: Uploading segments... Aug 7 21:45:33.044343424 (12h5m54.788400293s) [upload=12474]: Got next segment. Inline: true Aug 7 21:45:33.044349210 (12h5m54.78840607s) [upload=12474]: Waiting for error group managing segments... Aug 7 21:45:33.044391490 (12h5m54.78844834s) [upload=12474]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:33.044407486 (12h5m54.788464346s) [upload=12474]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:33.044413529 (12h5m54.788470389s) [upload=12474]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:33.198940222 (12h5m54.942997072s) [upload=12474]: Done uploading segments... Aug 7 21:45:33.198960825 (12h5m54.943017674s) [upload=12474]: Upload finished. err: <nil> Aug 7 21:45:57.787118330 (12h6m19.53117519s) [upload=12482]: Starting upload Aug 7 21:45:57.787191262 (12h6m19.531248122s) [upload=12482]: Done starting upload Aug 7 21:45:57.787268709 (12h6m19.53132556s) [upload=12482]: Uploading segments... Aug 7 21:45:57.787331847 (12h6m19.531388712s) [upload=12482]: Got next segment. Inline: true Aug 7 21:45:57.787339509 (12h6m19.53139637s) [upload=12482]: Waiting for error group managing segments... Aug 7 21:45:57.787386486 (12h6m19.531443346s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:57.787403824 (12h6m19.531460685s) [upload=12482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:57.787407765 (12h6m19.531464626s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:46:57.502016096 (12h7m19.246072946s) [upload=12482]: Done uploading segments... Aug 7 21:46:57.502068445 (12h7m19.246125296s) [upload=12482]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.084401018s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 5 out of 60 is failed (threshold=3) [....................XX..............XX.X....................] duration: 321.178403ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:53:36 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 321.203831ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXX......X.X........] duration: 3.102769534s FAIL (2025-08-07 21:53:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.102792095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...............] duration: 1.017984976s FAIL (2025-08-07 21:53:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.018026735s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [.......XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..............] duration: 907.072312ms FAIL (2025-08-07 21:53:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:41.354808211 (12h1m3.098865072s) [upload=12402]: Starting upload Aug 7 21:40:41.354938701 (12h1m3.098995568s) [upload=12402]: Done starting upload Aug 7 21:40:41.355036214 (12h1m3.099093064s) [upload=12402]: Uploading segments... Aug 7 21:40:41.355109185 (12h1m3.099166055s) [upload=12402]: Got next segment. Inline: true Aug 7 21:40:41.355116835 (12h1m3.0991737s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:40:41.355183407 (12h1m3.09924027s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:41.355190433 (12h1m3.099247297s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:41.355194357 (12h1m3.09925122s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:11.078938147 (12h1m32.822994999s) [upload=12402]: Done uploading segments... Aug 7 21:41:11.078958711 (12h1m32.823015562s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.899938086 (12h1m56.643994947s) [upload=12417]: Starting upload Aug 7 21:41:34.900014273 (12h1m56.644071133s) [upload=12417]: Done starting upload Aug 7 21:41:34.900081071 (12h1m56.644137926s) [upload=12417]: Uploading segments... Aug 7 21:41:34.900153322 (12h1m56.644210172s) [upload=12417]: Got next segment. Inline: true Aug 7 21:41:34.900158578 (12h1m56.644215438s) [upload=12417]: Waiting for error group managing segments... Aug 7 21:41:34.900228776 (12h1m56.644285623s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.900237327 (12h1m56.644294194s) [upload=12417]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.900241831 (12h1m56.644298696s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:04.627262793 (12h2m26.371319653s) [upload=12417]: Done uploading segments... Aug 7 21:42:04.627301743 (12h2m26.371358604s) [upload=12417]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:18.477725352 (12h2m40.221782248s) [upload=12428]: Starting upload Aug 7 21:42:18.477815072 (12h2m40.221871935s) [upload=12428]: Done starting upload Aug 7 21:42:18.477903321 (12h2m40.221960189s) [upload=12428]: Uploading segments... Aug 7 21:42:18.478047763 (12h2m40.222104633s) [upload=12428]: Got next segment. Inline: true Aug 7 21:42:18.478056185 (12h2m40.222113051s) [upload=12428]: Waiting for error group managing segments... Aug 7 21:42:18.478131478 (12h2m40.222188328s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:18.478138171 (12h2m40.222195021s) [upload=12428]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:18.478144895 (12h2m40.222201761s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:48.201994950 (12h3m9.946051801s) [upload=12428]: Done uploading segments... Aug 7 21:42:48.202028187 (12h3m9.946085036s) [upload=12428]: Upload finished. err: metaclient: context deadline exceeded Duration: 907.10818ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 924.730681ms FAIL (2025-08-07 21:53:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests Duration: 924.757832ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.......................XX.X..........X.....X........] duration: 3.042039352s FAIL (2025-08-07 21:53:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== piece_id=RXXWJ22ZFGJ42YRFAX4HRE7E2VPGAKWKOEBWWTBBEKXPLRQFF54A, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.931857584 (12h7m22.675914448s) [noise=true , address=104.36.82.39:4754 , piece_id=B64XNARGUDMJTQHTSEYX6LYDVYEPSFESYSHFJ2YATHOWE7V3JTSA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.932258428 (12h7m22.676315289s) [noise=true , address=104.36.82.55:4224 , piece_id=DVH3QKF4OWIMYY4EEMSNOIXW5CC54KN2M54CHUTU6YFC5MHRQFOA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.933368803 (12h7m22.677425673s) [noise=true , address=45.91.116.9:4288 , piece_id=ILYKFA6LDTNRBRGOVOR25H2VIGOUHQZQ3WPZDV3DZMZKH6ENKVNA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.933961999 (12h7m22.678018845s) [noise=true , address=104.36.82.43:4853 , piece_id=WBEOH7537PO55GJLYGC4YHMTKULN2T6BCCYAEVBRR5X4AQ5F4XSA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.934101417 (12h7m22.67815828s) [noise=true , address=104.36.82.35:4601 , piece_id=FIWLJE2E4X22266LQ44U7IBBZ4D7XLUGN6LRGENTOTC7H5WAIKMQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.934371093 (12h7m22.678427954s) [noise=true , address=104.36.82.44:4877 , piece_id=EDM3CTKQYKOSOP3JU3UUWN63LVG7BA6FGOAFF2IRKRNRUPK7GTVQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.934756333 (12h7m22.678813196s) [noise=true , address=104.36.82.61:4405 , piece_id=BA25J7JHA2KUSFTJDRPBPBHFVIUPTD7FE7IKSTJUEU3IGBMIB72A, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.934888956 (12h7m22.678945807s) [noise=true , address=64.49.13.154:4926 , piece_id=X5CLC7JS7PCLNSPLHTDA7VC4BXKIAZON3A2EVZCBZ3NZYYZD42HA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.935299805 (12h7m22.679356668s) [noise=true , address=104.36.82.9:4027 , piece_id=R3U7T2YCU7EKQULIM6KKZYBWQP2ZGNLHUUWQ7B4KJUJJIRAPZEPA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.935539944 (12h7m22.679596796s) [noise=true , address=64.49.13.169:4708 , piece_id=HTW7Y2VQQF5VDJPPLE265SZ7JU24T2KSEIUCSPEFQKZA2HNI2QWQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.935895434 (12h7m22.679952289s) [noise=true , address=104.36.82.36:4640 , piece_id=KCDKD2VKLEHD6V52CQBLYI4KRNDDMLDGGYBQPLX27JBDAIV64XQQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.936414259 (12h7m22.680471112s) [noise=true , address=104.36.82.13:4214 , piece_id=4X2ZSNKGKTJ4OZWJVFDYTR2RBRPOJRSOOVTXSBUPCIO3K53U5HJQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.936836219 (12h7m22.680893083s) [noise=true , address=104.36.82.62:4431 , piece_id=CRGUBEWHDRPMKGY47AYWHGCX4D3RKCP5OW6UV6TD77DIIVNBVZBA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.937535515 (12h7m22.68159243s) [noise=true , address=79.127.205.233:4642 , piece_id=YGV6RGALGLVITSWGUWQFJVQNCPHXWS67G33QWZZWAPIFE6RYQQSA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.937642268 (12h7m22.681699132s) [noise=true , address=104.36.82.58:4283 , piece_id=AEQQA4INRDBRLNAYRVDD4PSLOCBFM5HP4GOCWFOX47AHHJAEBWZA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.938008889 (12h7m22.682065754s) [noise=true , address=104.36.82.37:4688 , piece_id=LBVMHMZJFVI5HD3P33LOA2L5KA7EUB64TQWVFX3PSYUUSLCCLAPA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.938565671 (12h7m22.682622538s) [noise=true , address=104.36.82.23:4452 , piece_id=4XKVMVVLGCJRPKBC5HVJSX6M6UUI4QGFOCCKRHCMPNIOI3ULLWSA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:01.515934007 (12h7m23.259990877s) [upload=12498]: Next returned nil segment Aug 7 21:47:01.515957554 (12h7m23.260014424s) [upload=12498]: Waiting for error group managing segments... Aug 7 21:47:01.689221575 (12h7m23.433278437s) [noise=true , address=104.36.82.58:4283 , piece_id=AEQQA4INRDBRLNAYRVDD4PSLOCBFM5HP4GOCWFOX47AHHJAEBWZA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.696245173 (12h7m23.440302028s) [noise=true , address=104.36.82.45:4931 , piece_id=RXXWJ22ZFGJ42YRFAX4HRE7E2VPGAKWKOEBWWTBBEKXPLRQFF54A, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.698487716 (12h7m23.442544583s) [noise=true , address=104.36.82.23:4452 , piece_id=4XKVMVVLGCJRPKBC5HVJSX6M6UUI4QGFOCCKRHCMPNIOI3ULLWSA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.699902328 (12h7m23.443959181s) [noise=true , address=104.36.82.55:4224 , piece_id=DVH3QKF4OWIMYY4EEMSNOIXW5CC54KN2M54CHUTU6YFC5MHRQFOA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.704569768 (12h7m23.448626616s) [noise=true , address=104.36.82.62:4431 , piece_id=CRGUBEWHDRPMKGY47AYWHGCX4D3RKCP5OW6UV6TD77DIIVNBVZBA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.713265789 (12h7m23.457322661s) [noise=true , address=104.36.82.39:4754 , piece_id=B64XNARGUDMJTQHTSEYX6LYDVYEPSFESYSHFJ2YATHOWE7V3JTSA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.733750309 (12h7m23.477807164s) [noise=true , address=104.36.82.36:4640 , piece_id=KCDKD2VKLEHD6V52CQBLYI4KRNDDMLDGGYBQPLX27JBDAIV64XQQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.746947712 (12h7m23.491004568s) [noise=true , address=104.36.82.13:4214 , piece_id=4X2ZSNKGKTJ4OZWJVFDYTR2RBRPOJRSOOVTXSBUPCIO3K53U5HJQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.755976189 (12h7m23.500033045s) [noise=true , address=104.36.82.43:4853 , piece_id=WBEOH7537PO55GJLYGC4YHMTKULN2T6BCCYAEVBRR5X4AQ5F4XSA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.795757727 (12h7m23.539814619s) [noise=true , address=104.36.82.9:4027 , piece_id=R3U7T2YCU7EKQULIM6KKZYBWQP2ZGNLHUUWQ7B4KJUJJIRAPZEPA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.901256341 (12h7m23.645313207s) [noise=true , address=104.36.82.44:4877 , piece_id=EDM3CTKQYKOSOP3JU3UUWN63LVG7BA6FGOAFF2IRKRNRUPK7GTVQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.922821824 (12h7m23.666878693s) [noise=true , address=45.91.116.9:4288 , piece_id=ILYKFA6LDTNRBRGOVOR25H2VIGOUHQZQ3WPZDV3DZMZKH6ENKVNA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.950282380 (12h7m23.694339244s) [noise=true , address=104.36.82.37:4688 , piece_id=LBVMHMZJFVI5HD3P33LOA2L5KA7EUB64TQWVFX3PSYUUSLCCLAPA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:02.145128156 (12h7m23.889185029s) [noise=true , address=104.36.82.35:4601 , piece_id=FIWLJE2E4X22266LQ44U7IBBZ4D7XLUGN6LRGENTOTC7H5WAIKMQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:02.145174967 (12h7m23.889231834s) [seg_pos={0 0}, upload=12498]: Segment reached optimal threshold of 14 pieces. Aug 7 21:47:02.145375024 (12h7m23.889431872s) [noise=true , address=79.127.205.233:4642 , piece_id=YGV6RGALGLVITSWGUWQFJVQNCPHXWS67G33QWZZWAPIFE6RYQQSA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MYs8DF38z84JzwWUYRL7rxPZWNxfyH4WnUdT1L58BiLkQsUTW): protocol: expected piece hash; context canceled; EOF Aug 7 21:47:02.145801827 (12h7m23.889858706s) [noise=true , address=64.49.13.154:4926 , piece_id=X5CLC7JS7PCLNSPLHTDA7VC4BXKIAZON3A2EVZCBZ3NZYYZD42HA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bpEYLHxV7TM3KdheTwmwBxpPvm3PKccMVFLfLc5fTXtJZX5VY): protocol: expected piece hash; context canceled; EOF Aug 7 21:47:02.146021528 (12h7m23.890078399s) [noise=true , address=64.49.13.169:4708 , piece_id=HTW7Y2VQQF5VDJPPLE265SZ7JU24T2KSEIUCSPEFQKZA2HNI2QWQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: expected piece hash; context canceled; EOF Aug 7 21:47:02.146292773 (12h7m23.890349638s) [noise=true , address=104.36.82.61:4405 , piece_id=BA25J7JHA2KUSFTJDRPBPBHFVIUPTD7FE7IKSTJUEU3IGBMIB72A, seg_pos={0 0}, upload=12498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XPFXQaLxDzhZjcaACurTejzdA8S4szrXVJjc5pgJm9UMreNiy): protocol: read tcp 139.59.144.22:49992->104.36.82.61:4405: use of closed network connection; context canceled Aug 7 21:47:02.146329902 (12h7m23.890386769s) [seg_pos={0 0}, upload=12498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:47:02.146430465 (12h7m23.890487825s) [upload=12498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:47:02.146437380 (12h7m23.890494243s) [upload=12498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:02.446739950 (12h7m24.190796819s) [upload=12498]: Done uploading segments... Aug 7 21:47:02.446788220 (12h7m24.190845097s) [upload=12498]: Upload finished. err: <nil> Aug 7 21:47:52.318171063 (12h8m14.06222791s) [upload=12512]: Starting upload Aug 7 21:47:52.318240558 (12h8m14.062297427s) [upload=12512]: Done starting upload Aug 7 21:47:52.318352095 (12h8m14.062408942s) [upload=12512]: Uploading segments... Aug 7 21:47:52.318602540 (12h8m14.062659412s) [upload=12512]: Got next segment. Inline: false Aug 7 21:47:52.318662425 (12h8m14.062719297s) [upload=12512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:52.318669510 (12h8m14.062726381s) [upload=12512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:47:52.406892213 (12h8m14.150949079s) [upload=12512]: Next segment err: metaclient: Too Many Requests Aug 7 21:47:52.406918034 (12h8m14.150974895s) [upload=12512]: Done uploading segments... Aug 7 21:47:52.406934956 (12h8m14.150991807s) [upload=12512]: Upload finished. err: metaclient: Too Many Requests Duration: 3.042076673s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [....X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..............] duration: 2.721470771s FAIL (2025-08-07 21:53:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.721513091s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX......] duration: 2.47242101s FAIL (2025-08-07 21:53:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.472455508s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 40 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X.XX...........] duration: 885.753385ms FAIL (2025-08-07 21:53:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 885.788626ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXX......X.X.......] duration: 3.377769543s FAIL (2025-08-07 21:53:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.377798846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [..................X.XXXXXXXXXXXXXXXXXXXXXXXX......X.........] duration: 888.782847ms FAIL (2025-08-07 21:53:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 202662 (11h58m6.212259509s) [upload=12350]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:44.877004541 (11h58m6.621061411s) [upload=12363]: Starting upload Aug 7 21:37:44.877093129 (11h58m6.621150048s) [upload=12363]: Done starting upload Aug 7 21:37:44.877179828 (11h58m6.621236677s) [upload=12363]: Uploading segments... Aug 7 21:37:44.877253719 (11h58m6.621310589s) [upload=12363]: Got next segment. Inline: true Aug 7 21:37:44.877262492 (11h58m6.621319365s) [upload=12363]: Waiting for error group managing segments... Aug 7 21:37:44.877313668 (11h58m6.621370528s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:44.877319037 (11h58m6.621375897s) [upload=12363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:44.877322512 (11h58m6.621379375s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:44.603781017 (11h59m6.347837867s) [upload=12363]: Done uploading segments... Aug 7 21:38:44.603802220 (11h59m6.347859069s) [upload=12363]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:45.032933533 (11h59m6.776990379s) [upload=12375]: Starting upload Aug 7 21:38:45.033010670 (11h59m6.777067533s) [upload=12375]: Done starting upload Aug 7 21:38:45.033084965 (11h59m6.777141811s) [upload=12375]: Uploading segments... Aug 7 21:38:45.033156998 (11h59m6.777213868s) [upload=12375]: Got next segment. Inline: true Aug 7 21:38:45.033162279 (11h59m6.777219127s) [upload=12375]: Waiting for error group managing segments... Aug 7 21:38:45.033211860 (11h59m6.777268707s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:45.033216660 (11h59m6.77727352s) [upload=12375]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:45.033219329 (11h59m6.777276175s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:44.758881739 (12h0m6.502938587s) [upload=12375]: Done uploading segments... Aug 7 21:39:44.758902501 (12h0m6.502959346s) [upload=12375]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:45.185696259 (12h0m6.929753109s) [upload=12388]: Starting upload Aug 7 21:39:45.185761209 (12h0m6.929818072s) [upload=12388]: Done starting upload Aug 7 21:39:45.185834033 (12h0m6.929890896s) [upload=12388]: Uploading segments... Aug 7 21:39:45.185982126 (12h0m6.930038997s) [upload=12388]: Got next segment. Inline: true Aug 7 21:39:45.185990859 (12h0m6.930047726s) [upload=12388]: Waiting for error group managing segments... Aug 7 21:39:45.186067966 (12h0m6.930124827s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:45.186074972 (12h0m6.930131844s) [upload=12388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:45.186078706 (12h0m6.930135567s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:44.901948337 (12h1m6.646005185s) [upload=12388]: Done uploading segments... Aug 7 21:40:44.901969954 (12h1m6.646026803s) [upload=12388]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:53.544245995 (12h1m15.288302845s) [upload=12404]: Starting upload Aug 7 21:40:53.544357406 (12h1m15.288414275s) [upload=12404]: Done starting upload Aug 7 21:40:53.544447469 (12h1m15.288504323s) [upload=12404]: Uploading segments... Aug 7 21:40:53.544512878 (12h1m15.28856975s) [upload=12404]: Got next segment. Inline: true Aug 7 21:40:53.544519444 (12h1m15.288576314s) [upload=12404]: Waiting for error group managing segments... Aug 7 21:40:53.544579423 (12h1m15.288636288s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:53.544585620 (12h1m15.288642482s) [upload=12404]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:53.544589678 (12h1m15.288646542s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:05.856628151 (12h1m27.600685011s) [upload=12404]: Done uploading segments... Aug 7 21:41:05.856646351 (12h1m27.6007032s) [upload=12404]: Upload finished. err: <nil> Aug 7 21:41:43.946896616 (12h2m5.690953463s) [upload=12420]: Starting upload Aug 7 21:41:43.946973751 (12h2m5.69103062s) [upload=12420]: Done starting upload Aug 7 21:41:43.947068595 (12h2m5.691125457s) [upload=12420]: Uploading segments... Aug 7 21:41:43.947136055 (12h2m5.691192916s) [upload=12420]: Got next segment. Inline: true Aug 7 21:41:43.947141676 (12h2m5.691198537s) [upload=12420]: Waiting for error group managing segments... Aug 7 21:41:43.947184608 (12h2m5.691241458s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:43.947190015 (12h2m5.691246865s) [upload=12420]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:43.947193244 (12h2m5.691250106s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:31.804258579 (12h2m53.548315429s) [upload=12420]: Done uploading segments... Aug 7 21:42:31.804277273 (12h2m53.548334124s) [upload=12420]: Upload finished. err: <nil> Aug 7 21:42:52.266861509 (12h3m14.01091837s) [upload=12435]: Starting upload Aug 7 21:42:52.266938245 (12h3m14.010995105s) [upload=12435]: Done starting upload Aug 7 21:42:52.267013941 (12h3m14.011070791s) [upload=12435]: Uploading segments... Aug 7 21:42:52.267054220 (12h3m14.01111109s) [upload=12435]: Got next segment. Inline: true Aug 7 21:42:52.267059450 (12h3m14.011116311s) [upload=12435]: Waiting for error group managing segments... Aug 7 21:42:52.267099382 (12h3m14.011156243s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:52.267105199 (12h3m14.011162059s) [upload=12435]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:52.267108901 (12h3m14.011165762s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:01.858360925 (12h3m23.602417792s) [upload=12435]: Done uploading segments... Aug 7 21:43:01.858412136 (12h3m23.602468987s) [upload=12435]: Upload finished. err: <nil> Aug 7 21:43:32.197096129 (12h3m53.941152979s) [upload=12448]: Starting upload Aug 7 21:43:32.197186644 (12h3m53.941243495s) [upload=12448]: Done starting upload Aug 7 21:43:32.197270859 (12h3m53.941327709s) [upload=12448]: Uploading segments... Aug 7 21:43:32.197324582 (12h3m53.941381451s) [upload=12448]: Got next segment. Inline: true Aug 7 21:43:32.197329997 (12h3m53.941386847s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:43:32.197385019 (12h3m53.941441879s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:32.197390582 (12h3m53.941447443s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:32.197394073 (12h3m53.941450934s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:32.363120575 (12h3m54.107177435s) [upload=12448]: Done uploading segments... Aug 7 21:43:32.363140640 (12h3m54.107197486s) [upload=12448]: Upload finished. err: <nil> Aug 7 21:44:28.281971344 (12h4m50.026028222s) [upload=12459]: Starting upload Aug 7 21:44:28.282137259 (12h4m50.026194124s) [upload=12459]: Done starting upload Aug 7 21:44:28.282276848 (12h4m50.026333712s) [upload=12459]: Uploading segments... Aug 7 21:44:28.282367088 (12h4m50.026423964s) [upload=12459]: Got next segment. Inline: true Aug 7 21:44:28.282375073 (12h4m50.026431948s) [upload=12459]: Waiting for error group managing segments... Aug 7 21:44:28.282442525 (12h4m50.026499391s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:28.282449059 (12h4m50.026505928s) [upload=12459]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:28.282452578 (12h4m50.026509443s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:35.643697666 (12h4m57.387754517s) [upload=12459]: Done uploading segments... Aug 7 21:44:35.643716871 (12h4m57.387773725s) [upload=12459]: Upload finished. err: <nil> Aug 7 21:45:33.044130503 (12h5m54.788187353s) [upload=12474]: Starting upload Aug 7 21:45:33.044219677 (12h5m54.788276538s) [upload=12474]: Done starting upload Aug 7 21:45:33.044292138 (12h5m54.788348988s) [upload=12474]: Uploading segments... Aug 7 21:45:33.044343424 (12h5m54.788400293s) [upload=12474]: Got next segment. Inline: true Aug 7 21:45:33.044349210 (12h5m54.78840607s) [upload=12474]: Waiting for error group managing segments... Aug 7 21:45:33.044391490 (12h5m54.78844834s) [upload=12474]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:33.044407486 (12h5m54.788464346s) [upload=12474]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:33.044413529 (12h5m54.788470389s) [upload=12474]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:33.198940222 (12h5m54.942997072s) [upload=12474]: Done uploading segments... Aug 7 21:45:33.198960825 (12h5m54.943017674s) [upload=12474]: Upload finished. err: <nil> Aug 7 21:45:57.787118330 (12h6m19.53117519s) [upload=12482]: Starting upload Aug 7 21:45:57.787191262 (12h6m19.531248122s) [upload=12482]: Done starting upload Aug 7 21:45:57.787268709 (12h6m19.53132556s) [upload=12482]: Uploading segments... Aug 7 21:45:57.787331847 (12h6m19.531388712s) [upload=12482]: Got next segment. Inline: true Aug 7 21:45:57.787339509 (12h6m19.53139637s) [upload=12482]: Waiting for error group managing segments... Aug 7 21:45:57.787386486 (12h6m19.531443346s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:57.787403824 (12h6m19.531460685s) [upload=12482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:57.787407765 (12h6m19.531464626s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:46:57.502016096 (12h7m19.246072946s) [upload=12482]: Done uploading segments... Aug 7 21:46:57.502068445 (12h7m19.246125296s) [upload=12482]: Upload finished. err: metaclient: context deadline exceeded Duration: 888.839149ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXX....X..........] duration: 2.451794682s FAIL (2025-08-07 21:52:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== :35.590125129 (12h5m57.33418199s) [upload=12475]: Got next segment. Inline: false Aug 7 21:45:35.590176001 (12h5m57.33423285s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:35.590181518 (12h5m57.334238369s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:35.699013188 (12h5m57.443070052s) [seg_pos={0 0}, upload=12475]: Begin upload segment... Aug 7 21:45:35.699066084 (12h5m57.443122935s) [seg_pos={0 0}, upload=12475]: Done begin upload segment. Aug 7 21:45:35.699091982 (12h5m57.443148842s) [noise=true , address=77.90.31.17:4771 , piece_id=LDTJCTYFHCPL2SHCYONX3SJWBX3GME4GGILSOCHZWG67HKEAHDYQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.699635648 (12h5m57.443692518s) [noise=true , address=104.36.82.11:4115 , piece_id=U6HOQABV6JV5T4MFFT3EDQF6R77LXTPWZ33FJWUEYOWLWLPFLYWA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700048642 (12h5m57.444105509s) [noise=true , address=104.36.82.5:4840 , piece_id=V3VLVQPAAHDACW2WGUPYAGVDU2NRGR4FLQD2SFKVP2XLQALKP3SA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700358698 (12h5m57.444415561s) [noise=true , address=45.91.116.5:4299 , piece_id=YK7OVMQRE2PYUS2LDZOZGBKYV42U6VDWI2D6CMI4D2CIEV2DBACQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.700672646 (12h5m57.444729509s) [noise=true , address=45.91.116.14:4191 , piece_id=TYBU2DWWTVWWEEMTBD2PIQKVIU3YRHKXH7ACD7Y3VKZVDXPDRALA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701028296 (12h5m57.445085159s) [noise=true , address=104.36.82.13:4187 , piece_id=WCIN3NV6AFBYDUGO7TDPOMZG5YJ24ISXVB2WSLB46EZOPPGQKW2Q, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701400580 (12h5m57.44545744s) [noise=true , address=77.90.31.15:4697 , piece_id=XQ5CZFWWVUZ6MKSIYZVOUZG6YAA7UAVVISSKVAPJXXD7HPXJHK3A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.701701900 (12h5m57.445758763s) [noise=true , address=45.91.116.11:4119 , piece_id=K5FKXMYXUWHRKFZPDHFKZ425RAZLEQH254K65QOITKTC4APUYL7A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.702301822 (12h5m57.446358688s) [noise=true , address=45.91.116.11:4118 , piece_id=QPX3R57IYGZFNNOCPHOSQCRRQH6KJMDYSBPGHLX34DNABYYUXZ6A, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.702831318 (12h5m57.446888179s) [noise=true , address=104.36.82.10:4078 , piece_id=BEMP3VUCCLIJAY2OU2TFDJ5SQDJFO4OGL3UEHQ7YPMJGBRZ4E6BQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.703272004 (12h5m57.447328915s) [noise=true , address=45.91.116.6:4974 , piece_id=5HPCLRLKSQWUBMWQXW5LEOVQVBIKV62QAYF4YKQGQREWT3ZTAI2Q, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.703720399 (12h5m57.447777259s) [noise=true , address=104.36.82.3:4767 , piece_id=D3DBXBQ3UNQCWF5DR3HV46XZSOMRQHR33SQ6VSA7BXY7OZEWXNAA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704065340 (12h5m57.44812221s) [noise=true , address=77.90.31.13:4626 , piece_id=KM7OLI4GIFFIK4UTVRX26KMGVRZMASK47B7B7FOYCS6DU5SMZEXA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704350352 (12h5m57.448407212s) [noise=true , address=104.36.82.12:4151 , piece_id=UUTIELJOX7CUEFHHFWDD4XETR3EB7IQOCKUZXQALKKGXPBVWH4TA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.704707337 (12h5m57.448764198s) [noise=true , address=77.90.31.12:4590 , piece_id=BF63RJ2G676UQFZGWIAXFCZR2DC67F4AH5VZAB3MYIDDOMJKGISA, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:35.705111011 (12h5m57.449167878s) [noise=true , address=104.36.82.7:4911 , piece_id=6HSMRZBG26Y2XZ7NQZ6JD7P2BXZTDFDC6A5JIIS7SM7T6FAL2PWQ, seg_pos={0 0}, upload=12475]: Uploading piece... Aug 7 21:45:36.332909886 (12h5m58.076966752s) [upload=12475]: Next returned nil segment Aug 7 21:45:36.332933437 (12h5m58.076990311s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:45:36.489772849 (12h5m58.233829715s) [noise=true , address=104.36.82.10:4078 , piece_id=BEMP3VUCCLIJAY2OU2TFDJ5SQDJFO4OGL3UEHQ7YPMJGBRZ4E6BQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.509052197 (12h5m58.253109052s) [noise=true , address=104.36.82.12:4151 , piece_id=UUTIELJOX7CUEFHHFWDD4XETR3EB7IQOCKUZXQALKKGXPBVWH4TA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.542228341 (12h5m58.286285191s) [noise=true , address=77.90.31.12:4590 , piece_id=BF63RJ2G676UQFZGWIAXFCZR2DC67F4AH5VZAB3MYIDDOMJKGISA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.573060953 (12h5m58.317117805s) [noise=true , address=45.91.116.6:4974 , piece_id=5HPCLRLKSQWUBMWQXW5LEOVQVBIKV62QAYF4YKQGQREWT3ZTAI2Q, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.573821057 (12h5m58.317877923s) [noise=true , address=45.91.116.5:4299 , piece_id=YK7OVMQRE2PYUS2LDZOZGBKYV42U6VDWI2D6CMI4D2CIEV2DBACQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.620961481 (12h5m58.365018331s) [noise=true , address=77.90.31.17:4771 , piece_id=LDTJCTYFHCPL2SHCYONX3SJWBX3GME4GGILSOCHZWG67HKEAHDYQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.631043610 (12h5m58.375100483s) [noise=true , address=45.91.116.14:4191 , piece_id=TYBU2DWWTVWWEEMTBD2PIQKVIU3YRHKXH7ACD7Y3VKZVDXPDRALA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.632954530 (12h5m58.377011377s) [noise=true , address=45.91.116.11:4119 , piece_id=K5FKXMYXUWHRKFZPDHFKZ425RAZLEQH254K65QOITKTC4APUYL7A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.644557641 (12h5m58.388614491s) [noise=true , address=104.36.82.7:4911 , piece_id=6HSMRZBG26Y2XZ7NQZ6JD7P2BXZTDFDC6A5JIIS7SM7T6FAL2PWQ, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.649884441 (12h5m58.393941291s) [noise=true , address=77.90.31.13:4626 , piece_id=KM7OLI4GIFFIK4UTVRX26KMGVRZMASK47B7B7FOYCS6DU5SMZEXA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.661770778 (12h5m58.405827645s) [noise=true , address=77.90.31.15:4697 , piece_id=XQ5CZFWWVUZ6MKSIYZVOUZG6YAA7UAVVISSKVAPJXXD7HPXJHK3A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.725730770 (12h5m58.46978762s) [noise=true , address=104.36.82.3:4767 , piece_id=D3DBXBQ3UNQCWF5DR3HV46XZSOMRQHR33SQ6VSA7BXY7OZEWXNAA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.869176760 (12h5m58.613233628s) [noise=true , address=45.91.116.11:4118 , piece_id=QPX3R57IYGZFNNOCPHOSQCRRQH6KJMDYSBPGHLX34DNABYYUXZ6A, seg_pos={0 0}, upload=12475]: Done uploading piece. err: <nil> Aug 7 21:45:36.869206102 (12h5m58.61326296s) [seg_pos={0 0}, upload=12475]: Segment reached optimal threshold of 13 pieces. Aug 7 21:45:36.869561423 (12h5m58.613618286s) [noise=true , address=104.36.82.11:4115 , piece_id=U6HOQABV6JV5T4MFFT3EDQF6R77LXTPWZ33FJWUEYOWLWLPFLYWA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Aug 7 21:45:36.869806552 (12h5m58.613863414s) [noise=true , address=104.36.82.5:4840 , piece_id=V3VLVQPAAHDACW2WGUPYAGVDU2NRGR4FLQD2SFKVP2XLQALKP3SA, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection; write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36532->104.36.82.5:4840: use of closed network connection Aug 7 21:45:36.869995683 (12h5m58.614052547s) [noise=true , address=104.36.82.13:4187 , piece_id=WCIN3NV6AFBYDUGO7TDPOMZG5YJ24ISXVB2WSLB46EZOPPGQKW2Q, seg_pos={0 0}, upload=12475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39184->104.36.82.13:4187: use of closed network connection Aug 7 21:45:36.870032790 (12h5m58.614089658s) [seg_pos={0 0}, upload=12475]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:45:36.870148681 (12h5m58.614205541s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:45:36.870154041 (12h5m58.614210892s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:37.015978570 (12h5m58.76003542s) [upload=12475]: Done uploading segments... Aug 7 21:45:37.016019019 (12h5m58.760075869s) [upload=12475]: Upload finished. err: <nil> Aug 7 21:46:05.392535892 (12h6m27.136592742s) [upload=12486]: Starting upload Aug 7 21:46:05.392632482 (12h6m27.136689348s) [upload=12486]: Done starting upload Aug 7 21:46:05.392784466 (12h6m27.136841327s) [upload=12486]: Uploading segments... Aug 7 21:46:05.392963140 (12h6m27.13701999s) [upload=12486]: Got next segment. Inline: false Aug 7 21:46:05.393017249 (12h6m27.137074109s) [upload=12486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:05.393023723 (12h6m27.137080582s) [upload=12486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:46:35.116242044 (12h6m56.860298909s) [upload=12486]: Next segment err: context deadline exceeded Aug 7 21:46:35.116407347 (12h6m56.860464196s) [upload=12486]: Done uploading segments... Aug 7 21:46:35.116450315 (12h6m56.860507172s) [upload=12486]: Upload finished. err: context deadline exceeded Duration: 2.451831635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................XXXX.......] duration: 3.271430241s FAIL (2025-08-07 21:52:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== tchRequestItem_ObjectBegin Aug 7 21:24:45.813454815 (11h45m7.557511676s) [upload=12171]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:43.107142097 (11h46m4.851198968s) [seg_pos={0 0}, upload=12171]: Begin upload segment... Aug 7 21:25:43.107205391 (11h46m4.851262241s) [seg_pos={0 0}, upload=12171]: Done begin upload segment. Aug 7 21:25:43.107237087 (11h46m4.851293957s) [noise=true , address=104.36.82.15:4269 , piece_id=HTXIUN5HOU4B66MPMEXJJSUL6JNBE6FI6EB62IEG7P3WTZLAH47Q, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.107267905 (11h46m4.851324768s) [noise=true , address=104.36.82.8:4983 , piece_id=ZDR4NMXM2MJTWTLT5776GQIUXCIY7TMRNBQ43L73XOFFBLSA6KQQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.107830175 (11h46m4.851887046s) [noise=true , address=45.91.116.7:4013 , piece_id=JEWYLXASDWIZ6G3LSHWMUAFAZIKLWYO7KOZLEQEGNZB3HMNVSKYA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.108780038 (11h46m4.852836904s) [noise=true , address=104.36.82.16:4319 , piece_id=36CAJ2JJZF3CLN67PLA7VUM2F6NULTFSQ3WG6EDE734G3LNFZJTA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.108959605 (11h46m4.853016472s) [noise=true , address=104.36.82.43:4852 , piece_id=NBGNTQUWEOVY72D3XFFO74EFUL52BZP222Z6FNIYCFB43FOBIW2Q, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.109220359 (11h46m4.853277218s) [noise=true , address=79.127.201.214:4296 , piece_id=SK5JKFXZVF5XRW6STC5KDH5WNI7JHB6X5X5MLZ6WLLHG5SO6APSA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.109377214 (11h46m4.853434079s) [noise=true , address=104.36.82.37:4662 , piece_id=KZG6JYEDQUTQGQCTUAZK7R3JRD7GKFLKORVEC5LIZ47GVJ6MFZRA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.109626289 (11h46m4.853683159s) [noise=true , address=104.36.82.13:4195 , piece_id=YVPFOIDCG5F4LET7V5SZU2X6WKD4MCZWIQ6HUE7NVIOIVBRGDNHA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.109743295 (11h46m4.853800166s) [noise=true , address=79.127.205.245:4234 , piece_id=RADEPAKA2ZEQSI5UUX5BHRAR5JHNMOQI7MGZW42A4ZW52KJOXBRA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.110014430 (11h46m4.854071298s) [noise=true , address=64.49.13.138:4487 , piece_id=MXQL4B6PL3FOQS4WCCGCPF5DT5C3JDLYJNOEIE7GKNE5FRQJZ7EA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.110156657 (11h46m4.854213527s) [noise=true , address=45.91.116.8:4056 , piece_id=DOFJMEXQQBOTVMCWC77YLLDCUY4XYYRM3X5BOBSXXAPB7T4YVHZQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.110413863 (11h46m4.854470734s) [noise=true , address=104.36.82.35:4607 , piece_id=DQKF7V3EW6OT5C4IFITEQNOXUUHDOMG5AESWXUYJP3SERMWUF5VQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.110778464 (11h46m4.854835327s) [noise=true , address=104.36.82.38:4708 , piece_id=JUSE5AFFHKPHCDVDRN5WWWXQK6VHIY3CXQWMJ4EXEJXVMICMZIXA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.111117136 (11h46m4.855173999s) [noise=true , address=45.91.116.14:4197 , piece_id=52LI7AMEDE2OBW6TAIOP5Y2M4J4HAWEITEKIYPZZ7XQ2Z5CGYIBQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.111560812 (11h46m4.855617682s) [noise=true , address=104.36.82.35:4602 , piece_id=42ZEPU7YI52FBKWCUG2ZRDLPOLG5IYXSPR4HJ2QUSURO65WIAICQ, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.111976240 (11h46m4.856033107s) [noise=true , address=104.36.82.51:4073 , piece_id=HPC5WAIPHCOMTZDKPDPGZ7UJ43KPZZ2IFL5VFH7ILXHCF63D5NXA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.112141257 (11h46m4.856198127s) [noise=true , address=104.36.82.22:4511 , piece_id=WWW6R3JFLEQCGIAIZU6VXOC6XCPXFC3AWJ74PBL4PQXKTTSF535A, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.112560277 (11h46m4.856617148s) [noise=true , address=104.36.82.62:4412 , piece_id=6NAJYYYU3AX7KJNFBXTPEYRCFFTMV3ESLVFUWTEGKGNOEHFHJJKA, seg_pos={0 0}, upload=12171]: Uploading piece... Aug 7 21:25:43.675139110 (11h46m5.419195972s) [upload=12171]: Next returned nil segment Aug 7 21:25:43.675161049 (11h46m5.419217914s) [upload=12171]: Waiting for error group managing segments... Aug 7 21:25:43.872519019 (11h46m5.616575886s) [noise=true , address=104.36.82.62:4412 , piece_id=6NAJYYYU3AX7KJNFBXTPEYRCFFTMV3ESLVFUWTEGKGNOEHFHJJKA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.876126423 (11h46m5.62018327s) [noise=true , address=104.36.82.51:4073 , piece_id=HPC5WAIPHCOMTZDKPDPGZ7UJ43KPZZ2IFL5VFH7ILXHCF63D5NXA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.882439205 (11h46m5.626496076s) [noise=true , address=104.36.82.35:4602 , piece_id=42ZEPU7YI52FBKWCUG2ZRDLPOLG5IYXSPR4HJ2QUSURO65WIAICQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.886290836 (11h46m5.63034769s) [noise=true , address=104.36.82.22:4511 , piece_id=WWW6R3JFLEQCGIAIZU6VXOC6XCPXFC3AWJ74PBL4PQXKTTSF535A, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.888219892 (11h46m5.632276738s) [noise=true , address=104.36.82.15:4269 , piece_id=HTXIUN5HOU4B66MPMEXJJSUL6JNBE6FI6EB62IEG7P3WTZLAH47Q, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.888421317 (11h46m5.632478185s) [noise=true , address=45.91.116.14:4197 , piece_id=52LI7AMEDE2OBW6TAIOP5Y2M4J4HAWEITEKIYPZZ7XQ2Z5CGYIBQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.889366258 (11h46m5.633423104s) [noise=true , address=104.36.82.35:4607 , piece_id=DQKF7V3EW6OT5C4IFITEQNOXUUHDOMG5AESWXUYJP3SERMWUF5VQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.890288566 (11h46m5.634345437s) [noise=true , address=104.36.82.43:4852 , piece_id=NBGNTQUWEOVY72D3XFFO74EFUL52BZP222Z6FNIYCFB43FOBIW2Q, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.893016005 (11h46m5.637072875s) [noise=true , address=45.91.116.8:4056 , piece_id=DOFJMEXQQBOTVMCWC77YLLDCUY4XYYRM3X5BOBSXXAPB7T4YVHZQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.903527905 (11h46m5.647584771s) [noise=true , address=104.36.82.38:4708 , piece_id=JUSE5AFFHKPHCDVDRN5WWWXQK6VHIY3CXQWMJ4EXEJXVMICMZIXA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.912802006 (11h46m5.656858856s) [noise=true , address=45.91.116.7:4013 , piece_id=JEWYLXASDWIZ6G3LSHWMUAFAZIKLWYO7KOZLEQEGNZB3HMNVSKYA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:43.974200245 (11h46m5.718257092s) [noise=true , address=104.36.82.8:4983 , piece_id=ZDR4NMXM2MJTWTLT5776GQIUXCIY7TMRNBQ43L73XOFFBLSA6KQQ, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:44.128893332 (11h46m5.872950195s) [noise=true , address=79.127.201.214:4296 , piece_id=SK5JKFXZVF5XRW6STC5KDH5WNI7JHB6X5X5MLZ6WLLHG5SO6APSA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:44.143902251 (11h46m5.887959098s) [noise=true , address=104.36.82.37:4662 , piece_id=KZG6JYEDQUTQGQCTUAZK7R3JRD7GKFLKORVEC5LIZ47GVJ6MFZRA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: <nil> Aug 7 21:25:44.143991645 (11h46m5.888048495s) [seg_pos={0 0}, upload=12171]: Segment reached optimal threshold of 14 pieces. Aug 7 21:25:44.144122598 (11h46m5.888179458s) [noise=true , address=64.49.13.138:4487 , piece_id=MXQL4B6PL3FOQS4WCCGCPF5DT5C3JDLYJNOEIE7GKNE5FRQJZ7EA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF Aug 7 21:25:44.144194345 (11h46m5.888251205s) [noise=true , address=79.127.205.245:4234 , piece_id=RADEPAKA2ZEQSI5UUX5BHRAR5JHNMOQI7MGZW42A4ZW52KJOXBRA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJWq44d6y4Ai4tyZpCUvCgZeMHrVdiiSSngTR75YZZXbsgq2d): protocol: expected piece hash; context canceled; EOF Aug 7 21:25:44.144213145 (11h46m5.888269992s) [noise=true , address=104.36.82.13:4195 , piece_id=YVPFOIDCG5F4LET7V5SZU2X6WKD4MCZWIQ6HUE7NVIOIVBRGDNHA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH): protocol: write tcp 139.59.144.22:41808->104.36.82.13:4195: use of closed network connection; write tcp 139.59.144.22:41808->104.36.82.13:4195: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41808->104.36.82.13:4195: use of closed network connection Aug 7 21:25:44.144301767 (11h46m5.888358613s) [noise=true , address=104.36.82.16:4319 , piece_id=36CAJ2JJZF3CLN67PLA7VUM2F6NULTFSQ3WG6EDE734G3LNFZJTA, seg_pos={0 0}, upload=12171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hQTG2DzecaoN1Na2mY3JJqNj2qxtJDQkhMYmRmd9hve8D7T9C): protocol: write tcp 139.59.144.22:51894->104.36.82.16:4319: use of closed network connection; write tcp 139.59.144.22:51894->104.36.82.16:4319: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51894->104.36.82.16:4319: use of closed network connection Aug 7 21:25:44.144323393 (11h46m5.88838024s) [seg_pos={0 0}, upload=12171]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:25:44.144394152 (11h46m5.888450998s) [upload=12171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:25:44.144398730 (11h46m5.888455591s) [upload=12171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:47.207768845 (12h9m8.951825696s) [upload=12171]: Done uploading segments... Aug 7 21:48:47.207799297 (12h9m8.951856149s) [upload=12171]: Upload finished. err: metaclient: context deadline exceeded Duration:* 3.271531195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [........XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.............] duration: 962.546531ms FAIL (2025-08-07 21:52:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:41.354808211 (12h1m3.098865072s) [upload=12402]: Starting upload Aug 7 21:40:41.354938701 (12h1m3.098995568s) [upload=12402]: Done starting upload Aug 7 21:40:41.355036214 (12h1m3.099093064s) [upload=12402]: Uploading segments... Aug 7 21:40:41.355109185 (12h1m3.099166055s) [upload=12402]: Got next segment. Inline: true Aug 7 21:40:41.355116835 (12h1m3.0991737s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:40:41.355183407 (12h1m3.09924027s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:41.355190433 (12h1m3.099247297s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:41.355194357 (12h1m3.09925122s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:11.078938147 (12h1m32.822994999s) [upload=12402]: Done uploading segments... Aug 7 21:41:11.078958711 (12h1m32.823015562s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.899938086 (12h1m56.643994947s) [upload=12417]: Starting upload Aug 7 21:41:34.900014273 (12h1m56.644071133s) [upload=12417]: Done starting upload Aug 7 21:41:34.900081071 (12h1m56.644137926s) [upload=12417]: Uploading segments... Aug 7 21:41:34.900153322 (12h1m56.644210172s) [upload=12417]: Got next segment. Inline: true Aug 7 21:41:34.900158578 (12h1m56.644215438s) [upload=12417]: Waiting for error group managing segments... Aug 7 21:41:34.900228776 (12h1m56.644285623s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.900237327 (12h1m56.644294194s) [upload=12417]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.900241831 (12h1m56.644298696s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:04.627262793 (12h2m26.371319653s) [upload=12417]: Done uploading segments... Aug 7 21:42:04.627301743 (12h2m26.371358604s) [upload=12417]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:18.477725352 (12h2m40.221782248s) [upload=12428]: Starting upload Aug 7 21:42:18.477815072 (12h2m40.221871935s) [upload=12428]: Done starting upload Aug 7 21:42:18.477903321 (12h2m40.221960189s) [upload=12428]: Uploading segments... Aug 7 21:42:18.478047763 (12h2m40.222104633s) [upload=12428]: Got next segment. Inline: true Aug 7 21:42:18.478056185 (12h2m40.222113051s) [upload=12428]: Waiting for error group managing segments... Aug 7 21:42:18.478131478 (12h2m40.222188328s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:18.478138171 (12h2m40.222195021s) [upload=12428]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:18.478144895 (12h2m40.222201761s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:48.201994950 (12h3m9.946051801s) [upload=12428]: Done uploading segments... Aug 7 21:42:48.202028187 (12h3m9.946085036s) [upload=12428]: Upload finished. err: metaclient: context deadline exceeded Duration: 962.573068ms core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [......................X............X.X.X....................] duration: 105.772587ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:52:47 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 105.797268ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X.......................XX.X..........X.....X.......] duration: 3.025965347s FAIL (2025-08-07 21:52:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== piece_id=RXXWJ22ZFGJ42YRFAX4HRE7E2VPGAKWKOEBWWTBBEKXPLRQFF54A, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.931857584 (12h7m22.675914448s) [noise=true , address=104.36.82.39:4754 , piece_id=B64XNARGUDMJTQHTSEYX6LYDVYEPSFESYSHFJ2YATHOWE7V3JTSA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.932258428 (12h7m22.676315289s) [noise=true , address=104.36.82.55:4224 , piece_id=DVH3QKF4OWIMYY4EEMSNOIXW5CC54KN2M54CHUTU6YFC5MHRQFOA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.933368803 (12h7m22.677425673s) [noise=true , address=45.91.116.9:4288 , piece_id=ILYKFA6LDTNRBRGOVOR25H2VIGOUHQZQ3WPZDV3DZMZKH6ENKVNA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.933961999 (12h7m22.678018845s) [noise=true , address=104.36.82.43:4853 , piece_id=WBEOH7537PO55GJLYGC4YHMTKULN2T6BCCYAEVBRR5X4AQ5F4XSA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.934101417 (12h7m22.67815828s) [noise=true , address=104.36.82.35:4601 , piece_id=FIWLJE2E4X22266LQ44U7IBBZ4D7XLUGN6LRGENTOTC7H5WAIKMQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.934371093 (12h7m22.678427954s) [noise=true , address=104.36.82.44:4877 , piece_id=EDM3CTKQYKOSOP3JU3UUWN63LVG7BA6FGOAFF2IRKRNRUPK7GTVQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.934756333 (12h7m22.678813196s) [noise=true , address=104.36.82.61:4405 , piece_id=BA25J7JHA2KUSFTJDRPBPBHFVIUPTD7FE7IKSTJUEU3IGBMIB72A, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.934888956 (12h7m22.678945807s) [noise=true , address=64.49.13.154:4926 , piece_id=X5CLC7JS7PCLNSPLHTDA7VC4BXKIAZON3A2EVZCBZ3NZYYZD42HA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.935299805 (12h7m22.679356668s) [noise=true , address=104.36.82.9:4027 , piece_id=R3U7T2YCU7EKQULIM6KKZYBWQP2ZGNLHUUWQ7B4KJUJJIRAPZEPA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.935539944 (12h7m22.679596796s) [noise=true , address=64.49.13.169:4708 , piece_id=HTW7Y2VQQF5VDJPPLE265SZ7JU24T2KSEIUCSPEFQKZA2HNI2QWQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.935895434 (12h7m22.679952289s) [noise=true , address=104.36.82.36:4640 , piece_id=KCDKD2VKLEHD6V52CQBLYI4KRNDDMLDGGYBQPLX27JBDAIV64XQQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.936414259 (12h7m22.680471112s) [noise=true , address=104.36.82.13:4214 , piece_id=4X2ZSNKGKTJ4OZWJVFDYTR2RBRPOJRSOOVTXSBUPCIO3K53U5HJQ, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.936836219 (12h7m22.680893083s) [noise=true , address=104.36.82.62:4431 , piece_id=CRGUBEWHDRPMKGY47AYWHGCX4D3RKCP5OW6UV6TD77DIIVNBVZBA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.937535515 (12h7m22.68159243s) [noise=true , address=79.127.205.233:4642 , piece_id=YGV6RGALGLVITSWGUWQFJVQNCPHXWS67G33QWZZWAPIFE6RYQQSA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.937642268 (12h7m22.681699132s) [noise=true , address=104.36.82.58:4283 , piece_id=AEQQA4INRDBRLNAYRVDD4PSLOCBFM5HP4GOCWFOX47AHHJAEBWZA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.938008889 (12h7m22.682065754s) [noise=true , address=104.36.82.37:4688 , piece_id=LBVMHMZJFVI5HD3P33LOA2L5KA7EUB64TQWVFX3PSYUUSLCCLAPA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:00.938565671 (12h7m22.682622538s) [noise=true , address=104.36.82.23:4452 , piece_id=4XKVMVVLGCJRPKBC5HVJSX6M6UUI4QGFOCCKRHCMPNIOI3ULLWSA, seg_pos={0 0}, upload=12498]: Uploading piece... Aug 7 21:47:01.515934007 (12h7m23.259990877s) [upload=12498]: Next returned nil segment Aug 7 21:47:01.515957554 (12h7m23.260014424s) [upload=12498]: Waiting for error group managing segments... Aug 7 21:47:01.689221575 (12h7m23.433278437s) [noise=true , address=104.36.82.58:4283 , piece_id=AEQQA4INRDBRLNAYRVDD4PSLOCBFM5HP4GOCWFOX47AHHJAEBWZA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.696245173 (12h7m23.440302028s) [noise=true , address=104.36.82.45:4931 , piece_id=RXXWJ22ZFGJ42YRFAX4HRE7E2VPGAKWKOEBWWTBBEKXPLRQFF54A, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.698487716 (12h7m23.442544583s) [noise=true , address=104.36.82.23:4452 , piece_id=4XKVMVVLGCJRPKBC5HVJSX6M6UUI4QGFOCCKRHCMPNIOI3ULLWSA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.699902328 (12h7m23.443959181s) [noise=true , address=104.36.82.55:4224 , piece_id=DVH3QKF4OWIMYY4EEMSNOIXW5CC54KN2M54CHUTU6YFC5MHRQFOA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.704569768 (12h7m23.448626616s) [noise=true , address=104.36.82.62:4431 , piece_id=CRGUBEWHDRPMKGY47AYWHGCX4D3RKCP5OW6UV6TD77DIIVNBVZBA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.713265789 (12h7m23.457322661s) [noise=true , address=104.36.82.39:4754 , piece_id=B64XNARGUDMJTQHTSEYX6LYDVYEPSFESYSHFJ2YATHOWE7V3JTSA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.733750309 (12h7m23.477807164s) [noise=true , address=104.36.82.36:4640 , piece_id=KCDKD2VKLEHD6V52CQBLYI4KRNDDMLDGGYBQPLX27JBDAIV64XQQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.746947712 (12h7m23.491004568s) [noise=true , address=104.36.82.13:4214 , piece_id=4X2ZSNKGKTJ4OZWJVFDYTR2RBRPOJRSOOVTXSBUPCIO3K53U5HJQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.755976189 (12h7m23.500033045s) [noise=true , address=104.36.82.43:4853 , piece_id=WBEOH7537PO55GJLYGC4YHMTKULN2T6BCCYAEVBRR5X4AQ5F4XSA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.795757727 (12h7m23.539814619s) [noise=true , address=104.36.82.9:4027 , piece_id=R3U7T2YCU7EKQULIM6KKZYBWQP2ZGNLHUUWQ7B4KJUJJIRAPZEPA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.901256341 (12h7m23.645313207s) [noise=true , address=104.36.82.44:4877 , piece_id=EDM3CTKQYKOSOP3JU3UUWN63LVG7BA6FGOAFF2IRKRNRUPK7GTVQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.922821824 (12h7m23.666878693s) [noise=true , address=45.91.116.9:4288 , piece_id=ILYKFA6LDTNRBRGOVOR25H2VIGOUHQZQ3WPZDV3DZMZKH6ENKVNA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:01.950282380 (12h7m23.694339244s) [noise=true , address=104.36.82.37:4688 , piece_id=LBVMHMZJFVI5HD3P33LOA2L5KA7EUB64TQWVFX3PSYUUSLCCLAPA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:02.145128156 (12h7m23.889185029s) [noise=true , address=104.36.82.35:4601 , piece_id=FIWLJE2E4X22266LQ44U7IBBZ4D7XLUGN6LRGENTOTC7H5WAIKMQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: <nil> Aug 7 21:47:02.145174967 (12h7m23.889231834s) [seg_pos={0 0}, upload=12498]: Segment reached optimal threshold of 14 pieces. Aug 7 21:47:02.145375024 (12h7m23.889431872s) [noise=true , address=79.127.205.233:4642 , piece_id=YGV6RGALGLVITSWGUWQFJVQNCPHXWS67G33QWZZWAPIFE6RYQQSA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MYs8DF38z84JzwWUYRL7rxPZWNxfyH4WnUdT1L58BiLkQsUTW): protocol: expected piece hash; context canceled; EOF Aug 7 21:47:02.145801827 (12h7m23.889858706s) [noise=true , address=64.49.13.154:4926 , piece_id=X5CLC7JS7PCLNSPLHTDA7VC4BXKIAZON3A2EVZCBZ3NZYYZD42HA, seg_pos={0 0}, upload=12498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bpEYLHxV7TM3KdheTwmwBxpPvm3PKccMVFLfLc5fTXtJZX5VY): protocol: expected piece hash; context canceled; EOF Aug 7 21:47:02.146021528 (12h7m23.890078399s) [noise=true , address=64.49.13.169:4708 , piece_id=HTW7Y2VQQF5VDJPPLE265SZ7JU24T2KSEIUCSPEFQKZA2HNI2QWQ, seg_pos={0 0}, upload=12498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: expected piece hash; context canceled; EOF Aug 7 21:47:02.146292773 (12h7m23.890349638s) [noise=true , address=104.36.82.61:4405 , piece_id=BA25J7JHA2KUSFTJDRPBPBHFVIUPTD7FE7IKSTJUEU3IGBMIB72A, seg_pos={0 0}, upload=12498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XPFXQaLxDzhZjcaACurTejzdA8S4szrXVJjc5pgJm9UMreNiy): protocol: read tcp 139.59.144.22:49992->104.36.82.61:4405: use of closed network connection; context canceled Aug 7 21:47:02.146329902 (12h7m23.890386769s) [seg_pos={0 0}, upload=12498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:47:02.146430465 (12h7m23.890487825s) [upload=12498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:47:02.146437380 (12h7m23.890494243s) [upload=12498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:02.446739950 (12h7m24.190796819s) [upload=12498]: Done uploading segments... Aug 7 21:47:02.446788220 (12h7m24.190845097s) [upload=12498]: Upload finished. err: <nil> Aug 7 21:47:52.318171063 (12h8m14.06222791s) [upload=12512]: Starting upload Aug 7 21:47:52.318240558 (12h8m14.062297427s) [upload=12512]: Done starting upload Aug 7 21:47:52.318352095 (12h8m14.062408942s) [upload=12512]: Uploading segments... Aug 7 21:47:52.318602540 (12h8m14.062659412s) [upload=12512]: Got next segment. Inline: false Aug 7 21:47:52.318662425 (12h8m14.062719297s) [upload=12512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:52.318669510 (12h8m14.062726381s) [upload=12512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:47:52.406892213 (12h8m14.150949079s) [upload=12512]: Next segment err: metaclient: Too Many Requests Aug 7 21:47:52.406918034 (12h8m14.150974895s) [upload=12512]: Done uploading segments... Aug 7 21:47:52.406934956 (12h8m14.150991807s) [upload=12512]: Upload finished. err: metaclient: Too Many Requests Duration: 3.025996919s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.011196681s FAIL (2025-08-07 21:52:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests Duration: 2.011220457s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X..............] duration: 1.037177138s FAIL (2025-08-07 21:52:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.037196979s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [...................X.XXXXXXXXXXXXXXXXXXXXXXXX......X........] duration: 872.682495ms FAIL (2025-08-07 21:52:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 202662 (11h58m6.212259509s) [upload=12350]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:44.877004541 (11h58m6.621061411s) [upload=12363]: Starting upload Aug 7 21:37:44.877093129 (11h58m6.621150048s) [upload=12363]: Done starting upload Aug 7 21:37:44.877179828 (11h58m6.621236677s) [upload=12363]: Uploading segments... Aug 7 21:37:44.877253719 (11h58m6.621310589s) [upload=12363]: Got next segment. Inline: true Aug 7 21:37:44.877262492 (11h58m6.621319365s) [upload=12363]: Waiting for error group managing segments... Aug 7 21:37:44.877313668 (11h58m6.621370528s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:44.877319037 (11h58m6.621375897s) [upload=12363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:44.877322512 (11h58m6.621379375s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:44.603781017 (11h59m6.347837867s) [upload=12363]: Done uploading segments... Aug 7 21:38:44.603802220 (11h59m6.347859069s) [upload=12363]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:45.032933533 (11h59m6.776990379s) [upload=12375]: Starting upload Aug 7 21:38:45.033010670 (11h59m6.777067533s) [upload=12375]: Done starting upload Aug 7 21:38:45.033084965 (11h59m6.777141811s) [upload=12375]: Uploading segments... Aug 7 21:38:45.033156998 (11h59m6.777213868s) [upload=12375]: Got next segment. Inline: true Aug 7 21:38:45.033162279 (11h59m6.777219127s) [upload=12375]: Waiting for error group managing segments... Aug 7 21:38:45.033211860 (11h59m6.777268707s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:45.033216660 (11h59m6.77727352s) [upload=12375]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:45.033219329 (11h59m6.777276175s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:44.758881739 (12h0m6.502938587s) [upload=12375]: Done uploading segments... Aug 7 21:39:44.758902501 (12h0m6.502959346s) [upload=12375]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:45.185696259 (12h0m6.929753109s) [upload=12388]: Starting upload Aug 7 21:39:45.185761209 (12h0m6.929818072s) [upload=12388]: Done starting upload Aug 7 21:39:45.185834033 (12h0m6.929890896s) [upload=12388]: Uploading segments... Aug 7 21:39:45.185982126 (12h0m6.930038997s) [upload=12388]: Got next segment. Inline: true Aug 7 21:39:45.185990859 (12h0m6.930047726s) [upload=12388]: Waiting for error group managing segments... Aug 7 21:39:45.186067966 (12h0m6.930124827s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:45.186074972 (12h0m6.930131844s) [upload=12388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:45.186078706 (12h0m6.930135567s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:44.901948337 (12h1m6.646005185s) [upload=12388]: Done uploading segments... Aug 7 21:40:44.901969954 (12h1m6.646026803s) [upload=12388]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:53.544245995 (12h1m15.288302845s) [upload=12404]: Starting upload Aug 7 21:40:53.544357406 (12h1m15.288414275s) [upload=12404]: Done starting upload Aug 7 21:40:53.544447469 (12h1m15.288504323s) [upload=12404]: Uploading segments... Aug 7 21:40:53.544512878 (12h1m15.28856975s) [upload=12404]: Got next segment. Inline: true Aug 7 21:40:53.544519444 (12h1m15.288576314s) [upload=12404]: Waiting for error group managing segments... Aug 7 21:40:53.544579423 (12h1m15.288636288s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:53.544585620 (12h1m15.288642482s) [upload=12404]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:53.544589678 (12h1m15.288646542s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:05.856628151 (12h1m27.600685011s) [upload=12404]: Done uploading segments... Aug 7 21:41:05.856646351 (12h1m27.6007032s) [upload=12404]: Upload finished. err: <nil> Aug 7 21:41:43.946896616 (12h2m5.690953463s) [upload=12420]: Starting upload Aug 7 21:41:43.946973751 (12h2m5.69103062s) [upload=12420]: Done starting upload Aug 7 21:41:43.947068595 (12h2m5.691125457s) [upload=12420]: Uploading segments... Aug 7 21:41:43.947136055 (12h2m5.691192916s) [upload=12420]: Got next segment. Inline: true Aug 7 21:41:43.947141676 (12h2m5.691198537s) [upload=12420]: Waiting for error group managing segments... Aug 7 21:41:43.947184608 (12h2m5.691241458s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:43.947190015 (12h2m5.691246865s) [upload=12420]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:43.947193244 (12h2m5.691250106s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:31.804258579 (12h2m53.548315429s) [upload=12420]: Done uploading segments... Aug 7 21:42:31.804277273 (12h2m53.548334124s) [upload=12420]: Upload finished. err: <nil> Aug 7 21:42:52.266861509 (12h3m14.01091837s) [upload=12435]: Starting upload Aug 7 21:42:52.266938245 (12h3m14.010995105s) [upload=12435]: Done starting upload Aug 7 21:42:52.267013941 (12h3m14.011070791s) [upload=12435]: Uploading segments... Aug 7 21:42:52.267054220 (12h3m14.01111109s) [upload=12435]: Got next segment. Inline: true Aug 7 21:42:52.267059450 (12h3m14.011116311s) [upload=12435]: Waiting for error group managing segments... Aug 7 21:42:52.267099382 (12h3m14.011156243s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:52.267105199 (12h3m14.011162059s) [upload=12435]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:52.267108901 (12h3m14.011165762s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:01.858360925 (12h3m23.602417792s) [upload=12435]: Done uploading segments... Aug 7 21:43:01.858412136 (12h3m23.602468987s) [upload=12435]: Upload finished. err: <nil> Aug 7 21:43:32.197096129 (12h3m53.941152979s) [upload=12448]: Starting upload Aug 7 21:43:32.197186644 (12h3m53.941243495s) [upload=12448]: Done starting upload Aug 7 21:43:32.197270859 (12h3m53.941327709s) [upload=12448]: Uploading segments... Aug 7 21:43:32.197324582 (12h3m53.941381451s) [upload=12448]: Got next segment. Inline: true Aug 7 21:43:32.197329997 (12h3m53.941386847s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:43:32.197385019 (12h3m53.941441879s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:32.197390582 (12h3m53.941447443s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:32.197394073 (12h3m53.941450934s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:32.363120575 (12h3m54.107177435s) [upload=12448]: Done uploading segments... Aug 7 21:43:32.363140640 (12h3m54.107197486s) [upload=12448]: Upload finished. err: <nil> Aug 7 21:44:28.281971344 (12h4m50.026028222s) [upload=12459]: Starting upload Aug 7 21:44:28.282137259 (12h4m50.026194124s) [upload=12459]: Done starting upload Aug 7 21:44:28.282276848 (12h4m50.026333712s) [upload=12459]: Uploading segments... Aug 7 21:44:28.282367088 (12h4m50.026423964s) [upload=12459]: Got next segment. Inline: true Aug 7 21:44:28.282375073 (12h4m50.026431948s) [upload=12459]: Waiting for error group managing segments... Aug 7 21:44:28.282442525 (12h4m50.026499391s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:28.282449059 (12h4m50.026505928s) [upload=12459]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:28.282452578 (12h4m50.026509443s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:35.643697666 (12h4m57.387754517s) [upload=12459]: Done uploading segments... Aug 7 21:44:35.643716871 (12h4m57.387773725s) [upload=12459]: Upload finished. err: <nil> Aug 7 21:45:33.044130503 (12h5m54.788187353s) [upload=12474]: Starting upload Aug 7 21:45:33.044219677 (12h5m54.788276538s) [upload=12474]: Done starting upload Aug 7 21:45:33.044292138 (12h5m54.788348988s) [upload=12474]: Uploading segments... Aug 7 21:45:33.044343424 (12h5m54.788400293s) [upload=12474]: Got next segment. Inline: true Aug 7 21:45:33.044349210 (12h5m54.78840607s) [upload=12474]: Waiting for error group managing segments... Aug 7 21:45:33.044391490 (12h5m54.78844834s) [upload=12474]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:33.044407486 (12h5m54.788464346s) [upload=12474]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:33.044413529 (12h5m54.788470389s) [upload=12474]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:33.198940222 (12h5m54.942997072s) [upload=12474]: Done uploading segments... Aug 7 21:45:33.198960825 (12h5m54.943017674s) [upload=12474]: Upload finished. err: <nil> Aug 7 21:45:57.787118330 (12h6m19.53117519s) [upload=12482]: Starting upload Aug 7 21:45:57.787191262 (12h6m19.531248122s) [upload=12482]: Done starting upload Aug 7 21:45:57.787268709 (12h6m19.53132556s) [upload=12482]: Uploading segments... Aug 7 21:45:57.787331847 (12h6m19.531388712s) [upload=12482]: Got next segment. Inline: true Aug 7 21:45:57.787339509 (12h6m19.53139637s) [upload=12482]: Waiting for error group managing segments... Aug 7 21:45:57.787386486 (12h6m19.531443346s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:57.787403824 (12h6m19.531460685s) [upload=12482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:57.787407765 (12h6m19.531464626s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:46:57.502016096 (12h7m19.246072946s) [upload=12482]: Done uploading segments... Aug 7 21:46:57.502068445 (12h7m19.246125296s) [upload=12482]: Upload finished. err: metaclient: context deadline exceeded Duration: 872.712646ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.......................X............X.X.X...................] duration: 116.085609ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:52:31 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 116.11315ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 40 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X.XX..........] duration: 884.740213ms FAIL (2025-08-07 21:52:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 884.780571ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [.........XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX............] duration: 907.546434ms FAIL (2025-08-07 21:52:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:41.354808211 (12h1m3.098865072s) [upload=12402]: Starting upload Aug 7 21:40:41.354938701 (12h1m3.098995568s) [upload=12402]: Done starting upload Aug 7 21:40:41.355036214 (12h1m3.099093064s) [upload=12402]: Uploading segments... Aug 7 21:40:41.355109185 (12h1m3.099166055s) [upload=12402]: Got next segment. Inline: true Aug 7 21:40:41.355116835 (12h1m3.0991737s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:40:41.355183407 (12h1m3.09924027s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:41.355190433 (12h1m3.099247297s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:41.355194357 (12h1m3.09925122s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:11.078938147 (12h1m32.822994999s) [upload=12402]: Done uploading segments... Aug 7 21:41:11.078958711 (12h1m32.823015562s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.899938086 (12h1m56.643994947s) [upload=12417]: Starting upload Aug 7 21:41:34.900014273 (12h1m56.644071133s) [upload=12417]: Done starting upload Aug 7 21:41:34.900081071 (12h1m56.644137926s) [upload=12417]: Uploading segments... Aug 7 21:41:34.900153322 (12h1m56.644210172s) [upload=12417]: Got next segment. Inline: true Aug 7 21:41:34.900158578 (12h1m56.644215438s) [upload=12417]: Waiting for error group managing segments... Aug 7 21:41:34.900228776 (12h1m56.644285623s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.900237327 (12h1m56.644294194s) [upload=12417]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.900241831 (12h1m56.644298696s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:04.627262793 (12h2m26.371319653s) [upload=12417]: Done uploading segments... Aug 7 21:42:04.627301743 (12h2m26.371358604s) [upload=12417]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:18.477725352 (12h2m40.221782248s) [upload=12428]: Starting upload Aug 7 21:42:18.477815072 (12h2m40.221871935s) [upload=12428]: Done starting upload Aug 7 21:42:18.477903321 (12h2m40.221960189s) [upload=12428]: Uploading segments... Aug 7 21:42:18.478047763 (12h2m40.222104633s) [upload=12428]: Got next segment. Inline: true Aug 7 21:42:18.478056185 (12h2m40.222113051s) [upload=12428]: Waiting for error group managing segments... Aug 7 21:42:18.478131478 (12h2m40.222188328s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:18.478138171 (12h2m40.222195021s) [upload=12428]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:18.478144895 (12h2m40.222201761s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:48.201994950 (12h3m9.946051801s) [upload=12428]: Done uploading segments... Aug 7 21:42:48.202028187 (12h3m9.946085036s) [upload=12428]: Upload finished. err: metaclient: context deadline exceeded Duration: 907.576679ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)