Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
38 out of 60 is failed (threshold=3) [.....................XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001048523s
FAIL (2025-08-07 21:42:48 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:41.354808211 (12h1m3.098865072s) [upload=12402]: Starting upload Aug 7 21:40:41.354938701 (12h1m3.098995568s) [upload=12402]: Done starting upload Aug 7 21:40:41.355036214 (12h1m3.099093064s) [upload=12402]: Uploading segments... Aug 7 21:40:41.355109185 (12h1m3.099166055s) [upload=12402]: Got next segment. Inline: true Aug 7 21:40:41.355116835 (12h1m3.0991737s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:40:41.355183407 (12h1m3.09924027s) [upload=12402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:40:41.355190433 (12h1m3.099247297s) [upload=12402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:41.355194357 (12h1m3.09925122s) [upload=12402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:41:11.078938147 (12h1m32.822994999s) [upload=12402]: Done uploading segments... Aug 7 21:41:11.078958711 (12h1m32.823015562s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.899938086 (12h1m56.643994947s) [upload=12417]: Starting upload Aug 7 21:41:34.900014273 (12h1m56.644071133s) [upload=12417]: Done starting upload Aug 7 21:41:34.900081071 (12h1m56.644137926s) [upload=12417]: Uploading segments... Aug 7 21:41:34.900153322 (12h1m56.644210172s) [upload=12417]: Got next segment. Inline: true Aug 7 21:41:34.900158578 (12h1m56.644215438s) [upload=12417]: Waiting for error group managing segments... Aug 7 21:41:34.900228776 (12h1m56.644285623s) [upload=12417]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.900237327 (12h1m56.644294194s) [upload=12417]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.900241831 (12h1m56.644298696s) [upload=12417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:42:04.627262793 (12h2m26.371319653s) [upload=12417]: Done uploading segments... Aug 7 21:42:04.627301743 (12h2m26.371358604s) [upload=12417]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:18.477725352 (12h2m40.221782248s) [upload=12428]: Starting upload Aug 7 21:42:18.477815072 (12h2m40.221871935s) [upload=12428]: Done starting upload Aug 7 21:42:18.477903321 (12h2m40.221960189s) [upload=12428]: Uploading segments... Aug 7 21:42:18.478047763 (12h2m40.222104633s) [upload=12428]: Got next segment. Inline: true Aug 7 21:42:18.478056185 (12h2m40.222113051s) [upload=12428]: Waiting for error group managing segments... Aug 7 21:42:18.478131478 (12h2m40.222188328s) [upload=12428]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:42:18.478138171 (12h2m40.222195021s) [upload=12428]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:18.478144895 (12h2m40.222201761s) [upload=12428]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:42:48.201994950 (12h3m9.946051801s) [upload=12428]: Done uploading segments... Aug 7 21:42:48.202028187 (12h3m9.946085036s) [upload=12428]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.001076319s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
25 out of 60 is failed (threshold=3) [................................X.XXXXXXXXXXXXXXXXXXXXXXXX..] duration:
55.770778694s
FAIL (2025-08-07 21:42:39 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 21:30:21.398949225 (11h50m43.143006078s) [upload=12254]: Uploading segments... Aug 7 21:30:21.398993888 (11h50m43.143050757s) [upload=12254]: Got next segment. Inline: true Aug 7 21:30:21.399003181 (11h50m43.143060046s) [upload=12254]: Waiting for error group managing segments... Aug 7 21:30:21.399046484 (11h50m43.143103345s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:21.399052138 (11h50m43.143108999s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:21.399055744 (11h50m43.143112604s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:21.120945605 (11h51m42.865002833s) [upload=12254]: Done uploading segments... Aug 7 21:31:21.121002610 (11h51m42.865059457s) [upload=12254]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:21.580865540 (11h51m43.324922404s) [upload=12267]: Starting upload Aug 7 21:31:21.580962495 (11h51m43.32501936s) [upload=12267]: Done starting upload Aug 7 21:31:21.581042226 (11h51m43.325099087s) [upload=12267]: Uploading segments... Aug 7 21:31:21.581118360 (11h51m43.32517523s) [upload=12267]: Got next segment. Inline: true Aug 7 21:31:21.581124116 (11h51m43.325180981s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:31:21.581178813 (11h51m43.325235677s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:21.581184674 (11h51m43.325241538s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:21.581188187 (11h51m43.325245048s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:21.285242448 (11h52m43.029299295s) [upload=12267]: Done uploading segments... Aug 7 21:32:21.285264804 (11h52m43.029321651s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.849527306 (11h52m52.593584169s) [upload=12284]: Starting upload Aug 7 21:32:30.849593974 (11h52m52.593650837s) [upload=12284]: Done starting upload Aug 7 21:32:30.849656936 (11h52m52.593713781s) [upload=12284]: Uploading segments... Aug 7 21:32:30.849696837 (11h52m52.593753706s) [upload=12284]: Got next segment. Inline: true Aug 7 21:32:30.849701844 (11h52m52.593758708s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:32:30.849743987 (11h52m52.593800834s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.849748481 (11h52m52.59380534s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.849753457 (11h52m52.593810316s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.580274752 (11h53m52.3243316s) [upload=12284]: Done uploading segments... Aug 7 21:33:30.580297199 (11h53m52.324354049s) [upload=12284]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.996524574 (11h53m52.740581425s) [upload=12301]: Starting upload Aug 7 21:33:30.996601123 (11h53m52.740657983s) [upload=12301]: Done starting upload Aug 7 21:33:30.996679475 (11h53m52.740736329s) [upload=12301]: Uploading segments... Aug 7 21:33:30.996753641 (11h53m52.740810668s) [upload=12301]: Got next segment. Inline: true Aug 7 21:33:30.996760213 (11h53m52.740817074s) [upload=12301]: Waiting for error group managing segments... Aug 7 21:33:30.996825729 (11h53m52.740882584s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.996832674 (11h53m52.740889525s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.996838492 (11h53m52.740895343s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.718542438 (11h54m52.462599299s) [upload=12301]: Done uploading segments... Aug 7 21:34:30.718564899 (11h54m52.462621759s) [upload=12301]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.162032485 (11h54m52.906089348s) [upload=12316]: Starting upload Aug 7 21:34:31.162129373 (11h54m52.906186241s) [upload=12316]: Done starting upload Aug 7 21:34:31.162197010 (11h54m52.906253877s) [upload=12316]: Uploading segments... Aug 7 21:34:31.162265332 (11h54m52.906322201s) [upload=12316]: Got next segment. Inline: true Aug 7 21:34:31.162271192 (11h54m52.906328063s) [upload=12316]: Waiting for error group managing segments... Aug 7 21:34:31.162343122 (11h54m52.906399986s) [upload=12316]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.162350473 (11h54m52.906407351s) [upload=12316]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.162357235 (11h54m52.906414101s) [upload=12316]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.893978311 (11h55m52.638035172s) [upload=12316]: Done uploading segments... Aug 7 21:35:30.894001661 (11h55m52.638058513s) [upload=12316]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:44.593281832 (11h56m6.337338681s) [upload=12332]: Starting upload Aug 7 21:35:44.593379702 (11h56m6.337436562s) [upload=12332]: Done starting upload Aug 7 21:35:44.593440159 (11h56m6.33749701s) [upload=12332]: Uploading segments... Aug 7 21:35:44.593541679 (11h56m6.337598542s) [upload=12332]: Got next segment. Inline: true Aug 7 21:35:44.593548645 (11h56m6.337605505s) [upload=12332]: Waiting for error group managing segments... Aug 7 21:35:44.593622380 (11h56m6.337679229s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:44.593628916 (11h56m6.337685776s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:44.593632768 (11h56m6.337689686s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:44.320915363 (11h57m6.064972224s) [upload=12332]: Done uploading segments... Aug 7 21:36:44.320936819 (11h57m6.06499367s) [upload=12332]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:44.737096160 (11h57m6.48115302s) [upload=12350]: Starting upload Aug 7 21:36:44.737199233 (11h57m6.481256079s) [upload=12350]: Done starting upload Aug 7 21:36:44.737256356 (11h57m6.481313202s) [upload=12350]: Uploading segments... Aug 7 21:36:44.737309397 (11h57m6.481366267s) [upload=12350]: Got next segment. Inline: true Aug 7 21:36:44.737314124 (11h57m6.481370986s) [upload=12350]: Waiting for error group managing segments... Aug 7 21:36:44.737363580 (11h57m6.481420443s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:44.737369794 (11h57m6.481426664s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:44.737374614 (11h57m6.481431481s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:44.468181532 (11h58m6.212238379s) [upload=12350]: Done uploading segments... Aug 7 21:37:44.468202662 (11h58m6.212259509s) [upload=12350]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:44.877004541 (11h58m6.621061411s) [upload=12363]: Starting upload Aug 7 21:37:44.877093129 (11h58m6.621150048s) [upload=12363]: Done starting upload Aug 7 21:37:44.877179828 (11h58m6.621236677s) [upload=12363]: Uploading segments... Aug 7 21:37:44.877253719 (11h58m6.621310589s) [upload=12363]: Got next segment. Inline: true Aug 7 21:37:44.877262492 (11h58m6.621319365s) [upload=12363]: Waiting for error group managing segments... Aug 7 21:37:44.877313668 (11h58m6.621370528s) [upload=12363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:44.877319037 (11h58m6.621375897s) [upload=12363]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:44.877322512 (11h58m6.621379375s) [upload=12363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:44.603781017 (11h59m6.347837867s) [upload=12363]: Done uploading segments... Aug 7 21:38:44.603802220 (11h59m6.347859069s) [upload=12363]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:45.032933533 (11h59m6.776990379s) [upload=12375]: Starting upload Aug 7 21:38:45.033010670 (11h59m6.777067533s) [upload=12375]: Done starting upload Aug 7 21:38:45.033084965 (11h59m6.777141811s) [upload=12375]: Uploading segments... Aug 7 21:38:45.033156998 (11h59m6.777213868s) [upload=12375]: Got next segment. Inline: true Aug 7 21:38:45.033162279 (11h59m6.777219127s) [upload=12375]: Waiting for error group managing segments... Aug 7 21:38:45.033211860 (11h59m6.777268707s) [upload=12375]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:45.033216660 (11h59m6.77727352s) [upload=12375]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:45.033219329 (11h59m6.777276175s) [upload=12375]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:39:44.758881739 (12h0m6.502938587s) [upload=12375]: Done uploading segments... Aug 7 21:39:44.758902501 (12h0m6.502959346s) [upload=12375]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:45.185696259 (12h0m6.929753109s) [upload=12388]: Starting upload Aug 7 21:39:45.185761209 (12h0m6.929818072s) [upload=12388]: Done starting upload Aug 7 21:39:45.185834033 (12h0m6.929890896s) [upload=12388]: Uploading segments... Aug 7 21:39:45.185982126 (12h0m6.930038997s) [upload=12388]: Got next segment. Inline: true Aug 7 21:39:45.185990859 (12h0m6.930047726s) [upload=12388]: Waiting for error group managing segments... Aug 7 21:39:45.186067966 (12h0m6.930124827s) [upload=12388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:39:45.186074972 (12h0m6.930131844s) [upload=12388]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:45.186078706 (12h0m6.930135567s) [upload=12388]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:40:44.901948337 (12h1m6.646005185s) [upload=12388]: Done uploading segments... Aug 7 21:40:44.901969954 (12h1m6.646026803s) [upload=12388]: Upload finished. err: metaclient: context deadline exceeded
Duration: 55.770807034s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
5 out of 60 is failed (threshold=3) [.................................XX..............XX.X.......] duration:
352.746605ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:42:35 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 352.772491ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
26 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXX..] duration:
43.897893331s
FAIL (2025-08-07 21:42:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 43.897911401s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
4 out of 60 is failed (threshold=3) [......................................X............X.X.X....] duration:
117.307814ms remote:
"tcp:185.244.226.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:42:28 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 117.333899ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
36 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration:
30.001635697s
FAIL (2025-08-07 21:42:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001662403s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
37 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration:
923.884462ms
FAIL (2025-08-07 21:42:17 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 923.926487ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
857.261267ms
FAIL (2025-08-07 21:42:14 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests
Duration: 857.287037ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................X.......................XX.X..........] duration:
2.987062702s
FAIL (2025-08-07 21:42:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
979s) [noise=true , address=104.36.82.44:4892 , piece_id=W4NSH7UTQ34P5IZLD3XCALGBL6YZ5QJFVFFYVOF4DZYE4PLGC7EA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304266198 (11h54m5.048323067s) [noise=true , address=104.36.82.15:4261 , piece_id=ZN2CAPB5FV7XPEVJARIIFNPOPEY2GOEFQFRTZZNZLPLT765MRULA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304638399 (11h54m5.048695263s) [noise=true , address=104.36.82.4:4827 , piece_id=EGB2BTVBBNBDN57ZRD4WFTEYQ6GJCSUEKP3I6UI6XYEGFSWVQLWA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304964313 (11h54m5.049021182s) [noise=true , address=79.127.205.234:4585 , piece_id=JZO4TRJX7CFYUKI2U7QBY54FS3NRL2GG66FDTWAKHQIDQ6ZUWSFA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.305289194 (11h54m5.049346062s) [noise=true , address=104.36.82.19:4405 , piece_id=F2TDP4KIXX7SX6JX2FOKOILDTEDBSBC2Y4XB4HO7J6NZOSSTMXCQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.305663222 (11h54m5.049720088s) [noise=true , address=104.36.82.59:4303 , piece_id=HLXP2T7IJOJSVQTD6X4UJWZZ4E3HC6AXRGRTN2OUP4P3YXL4HBUQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.307375967 (11h54m5.051432827s) [noise=true , address=64.49.13.140:4506 , piece_id=Y7SSPSS7GYN7ZVI3JFV2KRPSPWKWUS44M6BPZKJCYD5NPBKZC7LA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.307994205 (11h54m5.052051068s) [noise=true , address=104.36.82.34:4578 , piece_id=WWMNXVJFSSU3AUT5DSWNAFTTZMNFK7COL6D6P2CEH4A4O73BRDJQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.308330311 (11h54m5.052387178s) [noise=true , address=104.36.82.45:4922 , piece_id=XKGSL3DS6JSY345IEK25AWBGSDX2IWUWDCDVXA66GBMPEO3WQWTQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.308696609 (11h54m5.052753472s) [noise=true , address=104.36.82.23:4462 , piece_id=CX6GMAOJFTZG2YUAQKRLCJYE2725ORXZL64IV34B2TIEPICFYXSQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.309027383 (11h54m5.053084243s) [noise=true , address=104.36.82.16:4297 , piece_id=Z5ACR62PJJNYADTFWU36W6L7GZF5NYSH7KT2J66UIY63NT3U3RHA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.309517639 (11h54m5.053574502s) [noise=true , address=104.36.82.3:4770 , piece_id=2DHQP37D3NH2DVJTM7QANBATIVW4KP7XAXLPTJHEQN3GBQGV4F2Q, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310005964 (11h54m5.054062831s) [noise=true , address=104.36.82.37:4666 , piece_id=ONYCF5TCS7H2NQNPK53UHS5HQORAZLVTCYNOFU7JAVREM5EQJSMA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310590144 (11h54m5.05464703s) [noise=true , address=104.36.82.60:4338 , piece_id=A6H2W5I3YT7NXCNCJTYZ4UFGIEDNZNRISA6RLHEMZOCUUPHYZYMQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310999480 (11h54m5.055056348s) [noise=true , address=45.91.116.11:4132 , piece_id=YL7AGZAI2AMUKWML22F64Z65AKJZR2Z3WOEUDZBNOK4IWRTSS4DQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311301215 (11h54m5.055358078s) [noise=true , address=79.127.205.233:4623 , piece_id=76LKINU73EN5LDGNSBLZRTLZCWHT45L25VUP4HWRPHO2ABV6CZDA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311594224 (11h54m5.055651092s) [noise=true , address=104.36.82.62:4417 , piece_id=JFFKIHB6YJZJLTQ5CLGATME7KA6B6TIML6XZCGYUQBFWRXAEB7DA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311902248 (11h54m5.055959119s) [noise=true , address=104.36.82.13:4189 , piece_id=RERQLVB7KU3V37Z6QVOJ5HGHV6IWXWOMJYPQMRBKLUM7ER3DUTCQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.887542534 (11h54m5.631599386s) [upload=12304]: Next returned nil segment
Aug 7 21:33:43.887567861 (11h54m5.631624821s) [upload=12304]: Waiting for error group managing segments...
Aug 7 21:33:44.048562222 (11h54m5.792619083s) [noise=true , address=104.36.82.37:4666 , piece_id=ONYCF5TCS7H2NQNPK53UHS5HQORAZLVTCYNOFU7JAVREM5EQJSMA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.059436730 (11h54m5.80349358s) [noise=true , address=104.36.82.19:4405 , piece_id=F2TDP4KIXX7SX6JX2FOKOILDTEDBSBC2Y4XB4HO7J6NZOSSTMXCQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.067299265 (11h54m5.811356118s) [noise=true , address=104.36.82.60:4338 , piece_id=A6H2W5I3YT7NXCNCJTYZ4UFGIEDNZNRISA6RLHEMZOCUUPHYZYMQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.113365621 (11h54m5.857422471s) [noise=true , address=104.36.82.13:4189 , piece_id=RERQLVB7KU3V37Z6QVOJ5HGHV6IWXWOMJYPQMRBKLUM7ER3DUTCQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.113990013 (11h54m5.858046868s) [noise=true , address=104.36.82.15:4261 , piece_id=ZN2CAPB5FV7XPEVJARIIFNPOPEY2GOEFQFRTZZNZLPLT765MRULA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.129055509 (11h54m5.873112362s) [noise=true , address=104.36.82.4:4827 , piece_id=EGB2BTVBBNBDN57ZRD4WFTEYQ6GJCSUEKP3I6UI6XYEGFSWVQLWA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.174268649 (11h54m5.918325499s) [noise=true , address=45.91.116.11:4132 , piece_id=YL7AGZAI2AMUKWML22F64Z65AKJZR2Z3WOEUDZBNOK4IWRTSS4DQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.212907217 (11h54m5.956964063s) [noise=true , address=104.36.82.59:4303 , piece_id=HLXP2T7IJOJSVQTD6X4UJWZZ4E3HC6AXRGRTN2OUP4P3YXL4HBUQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.234310642 (11h54m5.978367495s) [noise=true , address=104.36.82.62:4417 , piece_id=JFFKIHB6YJZJLTQ5CLGATME7KA6B6TIML6XZCGYUQBFWRXAEB7DA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.270648309 (11h54m6.014705171s) [noise=true , address=104.36.82.34:4578 , piece_id=WWMNXVJFSSU3AUT5DSWNAFTTZMNFK7COL6D6P2CEH4A4O73BRDJQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.470354111 (11h54m6.214410971s) [noise=true , address=104.36.82.3:4770 , piece_id=2DHQP37D3NH2DVJTM7QANBATIVW4KP7XAXLPTJHEQN3GBQGV4F2Q, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.489350040 (11h54m6.233406907s) [noise=true , address=104.36.82.44:4892 , piece_id=W4NSH7UTQ34P5IZLD3XCALGBL6YZ5QJFVFFYVOF4DZYE4PLGC7EA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.624667783 (11h54m6.36872463s) [noise=true , address=104.36.82.45:4922 , piece_id=XKGSL3DS6JSY345IEK25AWBGSDX2IWUWDCDVXA66GBMPEO3WQWTQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.641115000 (11h54m6.385171862s) [noise=true , address=79.127.205.233:4623 , piece_id=76LKINU73EN5LDGNSBLZRTLZCWHT45L25VUP4HWRPHO2ABV6CZDA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.641149015 (11h54m6.385205876s) [seg_pos={0 0}, upload=12304]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:33:44.641367964 (11h54m6.385424829s) [noise=true , address=64.49.13.140:4506 , piece_id=Y7SSPSS7GYN7ZVI3JFV2KRPSPWKWUS44M6BPZKJCYD5NPBKZC7LA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LYkWHxg4uodPG98gJ66uz34YjsBN9N3iiy6rWQpnLjhiKoCoW): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641595473 (11h54m6.385652338s) [noise=true , address=79.127.205.234:4585 , piece_id=JZO4TRJX7CFYUKI2U7QBY54FS3NRL2GG66FDTWAKHQIDQ6ZUWSFA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GHc2AyFWF13heMfJDetaST47dXeeYuq1hB3ZjYwN7vUfXuvBF): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641716595 (11h54m6.385773458s) [noise=true , address=104.36.82.23:4462 , piece_id=CX6GMAOJFTZG2YUAQKRLCJYE2725ORXZL64IV34B2TIEPICFYXSQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641795902 (11h54m6.385852754s) [noise=true , address=104.36.82.16:4297 , piece_id=Z5ACR62PJJNYADTFWU36W6L7GZF5NYSH7KT2J66UIY63NT3U3RHA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641820947 (11h54m6.385877796s) [seg_pos={0 0}, upload=12304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:33:44.641980472 (11h54m6.386037333s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:33:44.641986313 (11h54m6.386043173s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:33:44.811637442 (11h54m6.55569429s) [upload=12304]: Done uploading segments...
Aug 7 21:33:44.811656829 (11h54m6.555713675s) [upload=12304]: Upload finished. err: <nil>
Aug 7 21:34:01.932026219 (11h54m23.676083072s) [upload=12308]: Starting upload
Aug 7 21:34:01.932128481 (11h54m23.676185355s) [upload=12308]: Done starting upload
Aug 7 21:34:01.932434039 (11h54m23.676490888s) [upload=12308]: Uploading segments...
Aug 7 21:34:01.932820553 (11h54m23.676877417s) [upload=12308]: Got next segment. Inline: false
Aug 7 21:34:01.932925753 (11h54m23.676982622s) [upload=12308]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:34:01.932934357 (11h54m23.676991224s) [upload=12308]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:34:02.022671161 (11h54m23.766728061s) [upload=12308]: Next segment err: metaclient: Too Many Requests
Aug 7 21:34:02.022701989 (11h54m23.766758862s) [upload=12308]: Done uploading segments...
Aug 7 21:34:02.022759129 (11h54m23.766815993s) [upload=12308]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.987100985s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001831293s
FAIL (2025-08-07 21:42:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
ece...
Aug 7 21:39:34.919666330 (11h59m56.663723198s) [noise=true , address=45.91.116.13:4153 , piece_id=VPLNW4ELDDHLPEDQ2HVWPSIBC45XN7ORVLFG75FDALY7AFYRYNEQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.920138564 (11h59m56.664195435s) [noise=true , address=104.36.82.14:4222 , piece_id=3QK6KKHJKKEFLV424LSEVYXOP63K5MTQ3YDREYFKWXFZ7ZUEO2WA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.920195139 (11h59m56.664252015s) [noise=true , address=45.91.116.8:4048 , piece_id=WWF6JJIOTPJIKOBZXVC7FRVTDSIPLRIKNN3BDJABOPOBAT7GMHZQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.920822375 (11h59m56.664879235s) [noise=true , address=104.36.82.16:4294 , piece_id=OFVD5V2JJNQF6HGZDN7P2JXRNAAT6EWL67VD4HRPGM36SWKHI2CQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.920862650 (11h59m56.664919514s) [noise=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.921441470 (11h59m56.665498332s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.921487466 (11h59m56.665544328s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.922317331 (11h59m56.666374206s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.923026433 (11h59m56.667083296s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.923407595 (11h59m56.667464455s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.923763535 (11h59m56.667820401s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.924101773 (11h59m56.668158645s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.924425635 (11h59m56.668482498s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.924925232 (11h59m56.668982102s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.925233497 (11h59m56.669290362s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:35.531363219 (11h59m57.275420069s) [upload=12385]: Next returned nil segment
Aug 7 21:39:35.531379805 (11h59m57.275436664s) [upload=12385]: Waiting for error group managing segments...
Aug 7 21:39:35.709218321 (11h59m57.453275183s) [noise=true , address=104.36.82.16:4294 , piece_id=OFVD5V2JJNQF6HGZDN7P2JXRNAAT6EWL67VD4HRPGM36SWKHI2CQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.747949042 (11h59m57.492005889s) [noise=true , address=45.91.116.8:4048 , piece_id=WWF6JJIOTPJIKOBZXVC7FRVTDSIPLRIKNN3BDJABOPOBAT7GMHZQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.748418025 (11h59m57.492474871s) [noise=true , address=104.36.82.14:4222 , piece_id=3QK6KKHJKKEFLV424LSEVYXOP63K5MTQ3YDREYFKWXFZ7ZUEO2WA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.751190468 (11h59m57.495247318s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.754282840 (11h59m57.498339696s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.754668188 (11h59m57.498725035s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.760515970 (11h59m57.50457282s) [noise=true , address=45.91.116.13:4153 , piece_id=VPLNW4ELDDHLPEDQ2HVWPSIBC45XN7ORVLFG75FDALY7AFYRYNEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.765275771 (11h59m57.509332626s) [noise=true , address=77.90.31.18:4806 , piece_id=5VFMZTJ4UNYFVMRNXJUAAQXTOWT7GOJWM5LKVQYB4J2I6FWXY26Q, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.775261406 (11h59m57.519318257s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.794687407 (11h59m57.538744255s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.835050406 (11h59m57.579107253s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.835339286 (11h59m57.579396147s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.853563302 (11h59m57.597620154s) [noise=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.853590381 (11h59m57.597647242s) [seg_pos={0 0}, upload=12385]: Segment reached optimal threshold of 13 pieces.
Aug 7 21:39:35.853877651 (11h59m57.597934499s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Aug 7 21:39:35.854014198 (11h59m57.598071065s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Aug 7 21:39:35.854095786 (11h59m57.598152646s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection
Aug 7 21:39:35.854123369 (11h59m57.59818023s) [seg_pos={0 0}, upload=12385]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 21:39:35.854214014 (11h59m57.598270883s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:39:35.854220398 (11h59m57.598277266s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:41:05.070835032 (12h1m26.814891887s) [upload=12385]: Done uploading segments...
Aug 7 21:41:05.070862625 (12h1m26.814919475s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:41:05.504575464 (12h1m27.248632324s) [upload=12408]: Starting upload
Aug 7 21:41:05.504639224 (12h1m27.24869609s) [upload=12408]: Done starting upload
Aug 7 21:41:05.504795449 (12h1m27.248852297s) [upload=12408]: Uploading segments...
Aug 7 21:41:05.505027625 (12h1m27.249084472s) [upload=12408]: Got next segment. Inline: false
Aug 7 21:41:05.505067645 (12h1m27.249124506s) [upload=12408]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:41:05.505082107 (12h1m27.249138967s) [upload=12408]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:41:35.236785438 (12h1m56.980842303s) [upload=12408]: Next segment err: context deadline exceeded
Aug 7 21:41:35.236970291 (12h1m56.981027145s) [upload=12408]: Done uploading segments...
Aug 7 21:41:35.236995163 (12h1m56.981052013s) [upload=12408]: Upload finished. err: context deadline exceeded
Aug 7 21:41:35.644792324 (12h1m57.388849187s) [upload=12418]: Starting upload
Aug 7 21:41:35.644872275 (12h1m57.388929138s) [upload=12418]: Done starting upload
Aug 7 21:41:35.644961097 (12h1m57.389017947s) [upload=12418]: Uploading segments...
Aug 7 21:41:35.645252533 (12h1m57.389309384s) [upload=12418]: Got next segment. Inline: false
Aug 7 21:41:35.645298357 (12h1m57.389355205s) [upload=12418]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:41:35.645304084 (12h1m57.389360935s) [upload=12418]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:42:05.370939509 (12h2m27.114996372s) [upload=12418]: Next segment err: context deadline exceeded
Aug 7 21:42:05.370959077 (12h2m27.11501593s) [upload=12418]: Done uploading segments...
Aug 7 21:42:05.370976831 (12h2m27.115033685s) [upload=12418]: Upload finished. err: context deadline exceeded
Duration: 30.001850487s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
37 out of 60 is failed (threshold=3) [......................XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001313627s
FAIL (2025-08-07 21:42:04 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 21:33:27.128936415 (11h53m48.872993265s) [upload=12299]: Uploading segments... Aug 7 21:33:27.128981290 (11h53m48.873038156s) [upload=12299]: Got next segment. Inline: true Aug 7 21:33:27.128986811 (11h53m48.873043661s) [upload=12299]: Waiting for error group managing segments... Aug 7 21:33:27.129031120 (11h53m48.87308798s) [upload=12299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.129036464 (11h53m48.873093314s) [upload=12299]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.129041631 (11h53m48.873098481s) [upload=12299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:56.852338652 (11h54m18.596395503s) [upload=12299]: Done uploading segments... Aug 7 21:33:56.852369847 (11h54m18.596426695s) [upload=12299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:39.735661053 (11h55m1.479717916s) [upload=12318]: Starting upload Aug 7 21:34:39.735756008 (11h55m1.479812879s) [upload=12318]: Done starting upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:41.354808211 (12h1m3.098865072s) [upload=12402]: Starting upload Aug 7 21:40:41.354938701 (12h1m3.098995568s) [upload=12402]: Done starting upload Aug 7 21:40:41.355036214 (12h1m3.099093064s) [upload=12402]: Uploading segments... Aug 7 21:40:41.355109185 (12h1m3.099166055s) [upload=12402]: Got next segment. Inline: true Aug 7 21:40:41.355116835 (12h1m3.0991737s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:40:41.355183407 (12h1m3.09924027s) [upload=12402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:40:41.355190433 (12h1m3.099247297s) [upload=12402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:41.355194357 (12h1m3.09925122s) [upload=12402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:41:11.078938147 (12h1m32.822994999s) [upload=12402]: Done uploading segments... Aug 7 21:41:11.078958711 (12h1m32.823015562s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.899938086 (12h1m56.643994947s) [upload=12417]: Starting upload Aug 7 21:41:34.900014273 (12h1m56.644071133s) [upload=12417]: Done starting upload Aug 7 21:41:34.900081071 (12h1m56.644137926s) [upload=12417]: Uploading segments... Aug 7 21:41:34.900153322 (12h1m56.644210172s) [upload=12417]: Got next segment. Inline: true Aug 7 21:41:34.900158578 (12h1m56.644215438s) [upload=12417]: Waiting for error group managing segments... Aug 7 21:41:34.900228776 (12h1m56.644285623s) [upload=12417]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.900237327 (12h1m56.644294194s) [upload=12417]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.900241831 (12h1m56.644298696s) [upload=12417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:42:04.627262793 (12h2m26.371319653s) [upload=12417]: Done uploading segments... Aug 7 21:42:04.627301743 (12h2m26.371358604s) [upload=12417]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.001372668s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
976.70583ms
FAIL (2025-08-07 21:41:55 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests
Duration: 976.75121ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
4 out of 60 is failed (threshold=3) [.......................................X............X.X.X...] duration:
104.91683ms remote:
"tcp:185.244.226.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:41:54 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 104.956299ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
40 out of 60 is failed (threshold=3) [..................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000697043s
FAIL (2025-08-07 21:41:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000723349s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
26 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXX.] duration:
1m0.000848245s
FAIL (2025-08-07 21:41:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000875392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration:
30.001319438s
FAIL (2025-08-07 21:41:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
(11h59m42.675404653s) [upload=12385]: Done starting upload
Aug 7 21:39:20.931824296 (11h59m42.67588115s) [upload=12385]: Uploading segments...
Aug 7 21:39:20.931983801 (11h59m42.676040662s) [upload=12385]: Got next segment. Inline: false
Aug 7 21:39:20.932060613 (11h59m42.676117474s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:39:20.932067820 (11h59m42.676124683s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:39:34.918936632 (11h59m56.662993492s) [seg_pos={0 0}, upload=12385]: Begin upload segment...
Aug 7 21:39:34.919003504 (11h59m56.663060368s) [seg_pos={0 0}, upload=12385]: Done begin upload segment.
Aug 7 21:39:34.919035204 (11h59m56.663092067s) [noise=true , address=77.90.31.18:4806 , piece_id=5VFMZTJ4UNYFVMRNXJUAAQXTOWT7GOJWM5LKVQYB4J2I6FWXY26Q, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.919666330 (11h59m56.663723198s) [noise=true , address=45.91.116.13:4153 , piece_id=VPLNW4ELDDHLPEDQ2HVWPSIBC45XN7ORVLFG75FDALY7AFYRYNEQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.920138564 (11h59m56.664195435s) [noise=true , address=104.36.82.14:4222 , piece_id=3QK6KKHJKKEFLV424LSEVYXOP63K5MTQ3YDREYFKWXFZ7ZUEO2WA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.920195139 (11h59m56.664252015s) [noise=true , address=45.91.116.8:4048 , piece_id=WWF6JJIOTPJIKOBZXVC7FRVTDSIPLRIKNN3BDJABOPOBAT7GMHZQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.920822375 (11h59m56.664879235s) [noise=true , address=104.36.82.16:4294 , piece_id=OFVD5V2JJNQF6HGZDN7P2JXRNAAT6EWL67VD4HRPGM36SWKHI2CQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.920862650 (11h59m56.664919514s) [noise=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.921441470 (11h59m56.665498332s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.921487466 (11h59m56.665544328s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.922317331 (11h59m56.666374206s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.923026433 (11h59m56.667083296s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.923407595 (11h59m56.667464455s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.923763535 (11h59m56.667820401s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.924101773 (11h59m56.668158645s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.924425635 (11h59m56.668482498s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.924925232 (11h59m56.668982102s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.925233497 (11h59m56.669290362s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:35.531363219 (11h59m57.275420069s) [upload=12385]: Next returned nil segment
Aug 7 21:39:35.531379805 (11h59m57.275436664s) [upload=12385]: Waiting for error group managing segments...
Aug 7 21:39:35.709218321 (11h59m57.453275183s) [noise=true , address=104.36.82.16:4294 , piece_id=OFVD5V2JJNQF6HGZDN7P2JXRNAAT6EWL67VD4HRPGM36SWKHI2CQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.747949042 (11h59m57.492005889s) [noise=true , address=45.91.116.8:4048 , piece_id=WWF6JJIOTPJIKOBZXVC7FRVTDSIPLRIKNN3BDJABOPOBAT7GMHZQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.748418025 (11h59m57.492474871s) [noise=true , address=104.36.82.14:4222 , piece_id=3QK6KKHJKKEFLV424LSEVYXOP63K5MTQ3YDREYFKWXFZ7ZUEO2WA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.751190468 (11h59m57.495247318s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.754282840 (11h59m57.498339696s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.754668188 (11h59m57.498725035s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.760515970 (11h59m57.50457282s) [noise=true , address=45.91.116.13:4153 , piece_id=VPLNW4ELDDHLPEDQ2HVWPSIBC45XN7ORVLFG75FDALY7AFYRYNEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.765275771 (11h59m57.509332626s) [noise=true , address=77.90.31.18:4806 , piece_id=5VFMZTJ4UNYFVMRNXJUAAQXTOWT7GOJWM5LKVQYB4J2I6FWXY26Q, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.775261406 (11h59m57.519318257s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.794687407 (11h59m57.538744255s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.835050406 (11h59m57.579107253s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.835339286 (11h59m57.579396147s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.853563302 (11h59m57.597620154s) [noise=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.853590381 (11h59m57.597647242s) [seg_pos={0 0}, upload=12385]: Segment reached optimal threshold of 13 pieces.
Aug 7 21:39:35.853877651 (11h59m57.597934499s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Aug 7 21:39:35.854014198 (11h59m57.598071065s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Aug 7 21:39:35.854095786 (11h59m57.598152646s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection
Aug 7 21:39:35.854123369 (11h59m57.59818023s) [seg_pos={0 0}, upload=12385]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 21:39:35.854214014 (11h59m57.598270883s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:39:35.854220398 (11h59m57.598277266s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:41:05.070835032 (12h1m26.814891887s) [upload=12385]: Done uploading segments...
Aug 7 21:41:05.070862625 (12h1m26.814919475s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:41:05.504575464 (12h1m27.248632324s) [upload=12408]: Starting upload
Aug 7 21:41:05.504639224 (12h1m27.24869609s) [upload=12408]: Done starting upload
Aug 7 21:41:05.504795449 (12h1m27.248852297s) [upload=12408]: Uploading segments...
Aug 7 21:41:05.505027625 (12h1m27.249084472s) [upload=12408]: Got next segment. Inline: false
Aug 7 21:41:05.505067645 (12h1m27.249124506s) [upload=12408]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:41:05.505082107 (12h1m27.249138967s) [upload=12408]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:41:35.236785438 (12h1m56.980842303s) [upload=12408]: Next segment err: context deadline exceeded
Aug 7 21:41:35.236970291 (12h1m56.981027145s) [upload=12408]: Done uploading segments...
Aug 7 21:41:35.236995163 (12h1m56.981052013s) [upload=12408]: Upload finished. err: context deadline exceeded
Duration: 30.001334256s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
37 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000544743s
FAIL (2025-08-07 21:41:34 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000567521s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
35 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration:
15.103135301s
FAIL (2025-08-07 21:41:31 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.103156601s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
5 out of 60 is failed (threshold=3) [..................................XX..............XX.X......] duration:
2.500371533s remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:41:26 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 2.500399516s
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
3.203606298s
FAIL (2025-08-07 21:41:23 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests
Duration: 3.203635102s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
39 out of 60 is failed (threshold=3) [...................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001172469s
FAIL (2025-08-07 21:41:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001196202s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
25 out of 60 is failed (threshold=3) [.................................X.XXXXXXXXXXXXXXXXXXXXXXXX.] duration:
21.579878051s
FAIL (2025-08-07 21:41:14 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 21:30:21.398949225 (11h50m43.143006078s) [upload=12254]: Uploading segments... Aug 7 21:30:21.398993888 (11h50m43.143050757s) [upload=12254]: Got next segment. Inline: true Aug 7 21:30:21.399003181 (11h50m43.143060046s) [upload=12254]: Waiting for error group managing segments... Aug 7 21:30:21.399046484 (11h50m43.143103345s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:21.399052138 (11h50m43.143108999s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:21.399055744 (11h50m43.143112604s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:21.120945605 (11h51m42.865002833s) [upload=12254]: Done uploading segments... Aug 7 21:31:21.121002610 (11h51m42.865059457s) [upload=12254]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:21.580865540 (11h51m43.324922404s) [upload=12267]: Starting upload Aug 7 21:31:21.580962495 (11h51m43.32501936s) [upload=12267]: Done starting upload Aug 7 21:31:21.581042226 (11h51m43.325099087s) [upload=12267]: Uploading segments... Aug 7 21:31:21.581118360 (11h51m43.32517523s) [upload=12267]: Got next segment. Inline: true Aug 7 21:31:21.581124116 (11h51m43.325180981s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:31:21.581178813 (11h51m43.325235677s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:21.581184674 (11h51m43.325241538s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:21.581188187 (11h51m43.325245048s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:21.285242448 (11h52m43.029299295s) [upload=12267]: Done uploading segments... Aug 7 21:32:21.285264804 (11h52m43.029321651s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.849527306 (11h52m52.593584169s) [upload=12284]: Starting upload Aug 7 21:32:30.849593974 (11h52m52.593650837s) [upload=12284]: Done starting upload Aug 7 21:32:30.849656936 (11h52m52.593713781s) [upload=12284]: Uploading segments... Aug 7 21:32:30.849696837 (11h52m52.593753706s) [upload=12284]: Got next segment. Inline: true Aug 7 21:32:30.849701844 (11h52m52.593758708s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:32:30.849743987 (11h52m52.593800834s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.849748481 (11h52m52.59380534s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.849753457 (11h52m52.593810316s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.580274752 (11h53m52.3243316s) [upload=12284]: Done uploading segments... Aug 7 21:33:30.580297199 (11h53m52.324354049s) [upload=12284]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.996524574 (11h53m52.740581425s) [upload=12301]: Starting upload Aug 7 21:33:30.996601123 (11h53m52.740657983s) [upload=12301]: Done starting upload Aug 7 21:33:30.996679475 (11h53m52.740736329s) [upload=12301]: Uploading segments... Aug 7 21:33:30.996753641 (11h53m52.740810668s) [upload=12301]: Got next segment. Inline: true Aug 7 21:33:30.996760213 (11h53m52.740817074s) [upload=12301]: Waiting for error group managing segments... Aug 7 21:33:30.996825729 (11h53m52.740882584s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.996832674 (11h53m52.740889525s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.996838492 (11h53m52.740895343s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.718542438 (11h54m52.462599299s) [upload=12301]: Done uploading segments... Aug 7 21:34:30.718564899 (11h54m52.462621759s) [upload=12301]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.162032485 (11h54m52.906089348s) [upload=12316]: Starting upload Aug 7 21:34:31.162129373 (11h54m52.906186241s) [upload=12316]: Done starting upload Aug 7 21:34:31.162197010 (11h54m52.906253877s) [upload=12316]: Uploading segments... Aug 7 21:34:31.162265332 (11h54m52.906322201s) [upload=12316]: Got next segment. Inline: true Aug 7 21:34:31.162271192 (11h54m52.906328063s) [upload=12316]: Waiting for error group managing segments... Aug 7 21:34:31.162343122 (11h54m52.906399986s) [upload=12316]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.162350473 (11h54m52.906407351s) [upload=12316]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.162357235 (11h54m52.906414101s) [upload=12316]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.893978311 (11h55m52.638035172s) [upload=12316]: Done uploading segments... Aug 7 21:35:30.894001661 (11h55m52.638058513s) [upload=12316]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:44.593281832 (11h56m6.337338681s) [upload=12332]: Starting upload Aug 7 21:35:44.593379702 (11h56m6.337436562s) [upload=12332]: Done starting upload Aug 7 21:35:44.593440159 (11h56m6.33749701s) [upload=12332]: Uploading segments... Aug 7 21:35:44.593541679 (11h56m6.337598542s) [upload=12332]: Got next segment. Inline: true Aug 7 21:35:44.593548645 (11h56m6.337605505s) [upload=12332]: Waiting for error group managing segments... Aug 7 21:35:44.593622380 (11h56m6.337679229s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:44.593628916 (11h56m6.337685776s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:44.593632768 (11h56m6.337689686s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:44.320915363 (11h57m6.064972224s) [upload=12332]: Done uploading segments... Aug 7 21:36:44.320936819 (11h57m6.06499367s) [upload=12332]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:44.737096160 (11h57m6.48115302s) [upload=12350]: Starting upload Aug 7 21:36:44.737199233 (11h57m6.481256079s) [upload=12350]: Done starting upload Aug 7 21:36:44.737256356 (11h57m6.481313202s) [upload=12350]: Uploading segments... Aug 7 21:36:44.737309397 (11h57m6.481366267s) [upload=12350]: Got next segment. Inline: true Aug 7 21:36:44.737314124 (11h57m6.481370986s) [upload=12350]: Waiting for error group managing segments... Aug 7 21:36:44.737363580 (11h57m6.481420443s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:44.737369794 (11h57m6.481426664s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:44.737374614 (11h57m6.481431481s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:44.468181532 (11h58m6.212238379s) [upload=12350]: Done uploading segments... Aug 7 21:37:44.468202662 (11h58m6.212259509s) [upload=12350]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:44.877004541 (11h58m6.621061411s) [upload=12363]: Starting upload Aug 7 21:37:44.877093129 (11h58m6.621150048s) [upload=12363]: Done starting upload Aug 7 21:37:44.877179828 (11h58m6.621236677s) [upload=12363]: Uploading segments... Aug 7 21:37:44.877253719 (11h58m6.621310589s) [upload=12363]: Got next segment. Inline: true Aug 7 21:37:44.877262492 (11h58m6.621319365s) [upload=12363]: Waiting for error group managing segments... Aug 7 21:37:44.877313668 (11h58m6.621370528s) [upload=12363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:44.877319037 (11h58m6.621375897s) [upload=12363]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:44.877322512 (11h58m6.621379375s) [upload=12363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:44.603781017 (11h59m6.347837867s) [upload=12363]: Done uploading segments... Aug 7 21:38:44.603802220 (11h59m6.347859069s) [upload=12363]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:45.032933533 (11h59m6.776990379s) [upload=12375]: Starting upload Aug 7 21:38:45.033010670 (11h59m6.777067533s) [upload=12375]: Done starting upload Aug 7 21:38:45.033084965 (11h59m6.777141811s) [upload=12375]: Uploading segments... Aug 7 21:38:45.033156998 (11h59m6.777213868s) [upload=12375]: Got next segment. Inline: true Aug 7 21:38:45.033162279 (11h59m6.777219127s) [upload=12375]: Waiting for error group managing segments... Aug 7 21:38:45.033211860 (11h59m6.777268707s) [upload=12375]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:45.033216660 (11h59m6.77727352s) [upload=12375]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:45.033219329 (11h59m6.777276175s) [upload=12375]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:39:44.758881739 (12h0m6.502938587s) [upload=12375]: Done uploading segments... Aug 7 21:39:44.758902501 (12h0m6.502959346s) [upload=12375]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:45.185696259 (12h0m6.929753109s) [upload=12388]: Starting upload Aug 7 21:39:45.185761209 (12h0m6.929818072s) [upload=12388]: Done starting upload Aug 7 21:39:45.185834033 (12h0m6.929890896s) [upload=12388]: Uploading segments... Aug 7 21:39:45.185982126 (12h0m6.930038997s) [upload=12388]: Got next segment. Inline: true Aug 7 21:39:45.185990859 (12h0m6.930047726s) [upload=12388]: Waiting for error group managing segments... Aug 7 21:39:45.186067966 (12h0m6.930124827s) [upload=12388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:39:45.186074972 (12h0m6.930131844s) [upload=12388]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:45.186078706 (12h0m6.930135567s) [upload=12388]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:40:44.901948337 (12h1m6.646005185s) [upload=12388]: Done uploading segments... Aug 7 21:40:44.901969954 (12h1m6.646026803s) [upload=12388]: Upload finished. err: metaclient: context deadline exceeded
Duration: 21.579907582s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [X......................X.......................XX.X.........] duration:
2.830233541s
FAIL (2025-08-07 21:41:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
979s) [noise=true , address=104.36.82.44:4892 , piece_id=W4NSH7UTQ34P5IZLD3XCALGBL6YZ5QJFVFFYVOF4DZYE4PLGC7EA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304266198 (11h54m5.048323067s) [noise=true , address=104.36.82.15:4261 , piece_id=ZN2CAPB5FV7XPEVJARIIFNPOPEY2GOEFQFRTZZNZLPLT765MRULA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304638399 (11h54m5.048695263s) [noise=true , address=104.36.82.4:4827 , piece_id=EGB2BTVBBNBDN57ZRD4WFTEYQ6GJCSUEKP3I6UI6XYEGFSWVQLWA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304964313 (11h54m5.049021182s) [noise=true , address=79.127.205.234:4585 , piece_id=JZO4TRJX7CFYUKI2U7QBY54FS3NRL2GG66FDTWAKHQIDQ6ZUWSFA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.305289194 (11h54m5.049346062s) [noise=true , address=104.36.82.19:4405 , piece_id=F2TDP4KIXX7SX6JX2FOKOILDTEDBSBC2Y4XB4HO7J6NZOSSTMXCQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.305663222 (11h54m5.049720088s) [noise=true , address=104.36.82.59:4303 , piece_id=HLXP2T7IJOJSVQTD6X4UJWZZ4E3HC6AXRGRTN2OUP4P3YXL4HBUQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.307375967 (11h54m5.051432827s) [noise=true , address=64.49.13.140:4506 , piece_id=Y7SSPSS7GYN7ZVI3JFV2KRPSPWKWUS44M6BPZKJCYD5NPBKZC7LA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.307994205 (11h54m5.052051068s) [noise=true , address=104.36.82.34:4578 , piece_id=WWMNXVJFSSU3AUT5DSWNAFTTZMNFK7COL6D6P2CEH4A4O73BRDJQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.308330311 (11h54m5.052387178s) [noise=true , address=104.36.82.45:4922 , piece_id=XKGSL3DS6JSY345IEK25AWBGSDX2IWUWDCDVXA66GBMPEO3WQWTQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.308696609 (11h54m5.052753472s) [noise=true , address=104.36.82.23:4462 , piece_id=CX6GMAOJFTZG2YUAQKRLCJYE2725ORXZL64IV34B2TIEPICFYXSQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.309027383 (11h54m5.053084243s) [noise=true , address=104.36.82.16:4297 , piece_id=Z5ACR62PJJNYADTFWU36W6L7GZF5NYSH7KT2J66UIY63NT3U3RHA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.309517639 (11h54m5.053574502s) [noise=true , address=104.36.82.3:4770 , piece_id=2DHQP37D3NH2DVJTM7QANBATIVW4KP7XAXLPTJHEQN3GBQGV4F2Q, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310005964 (11h54m5.054062831s) [noise=true , address=104.36.82.37:4666 , piece_id=ONYCF5TCS7H2NQNPK53UHS5HQORAZLVTCYNOFU7JAVREM5EQJSMA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310590144 (11h54m5.05464703s) [noise=true , address=104.36.82.60:4338 , piece_id=A6H2W5I3YT7NXCNCJTYZ4UFGIEDNZNRISA6RLHEMZOCUUPHYZYMQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310999480 (11h54m5.055056348s) [noise=true , address=45.91.116.11:4132 , piece_id=YL7AGZAI2AMUKWML22F64Z65AKJZR2Z3WOEUDZBNOK4IWRTSS4DQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311301215 (11h54m5.055358078s) [noise=true , address=79.127.205.233:4623 , piece_id=76LKINU73EN5LDGNSBLZRTLZCWHT45L25VUP4HWRPHO2ABV6CZDA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311594224 (11h54m5.055651092s) [noise=true , address=104.36.82.62:4417 , piece_id=JFFKIHB6YJZJLTQ5CLGATME7KA6B6TIML6XZCGYUQBFWRXAEB7DA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311902248 (11h54m5.055959119s) [noise=true , address=104.36.82.13:4189 , piece_id=RERQLVB7KU3V37Z6QVOJ5HGHV6IWXWOMJYPQMRBKLUM7ER3DUTCQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.887542534 (11h54m5.631599386s) [upload=12304]: Next returned nil segment
Aug 7 21:33:43.887567861 (11h54m5.631624821s) [upload=12304]: Waiting for error group managing segments...
Aug 7 21:33:44.048562222 (11h54m5.792619083s) [noise=true , address=104.36.82.37:4666 , piece_id=ONYCF5TCS7H2NQNPK53UHS5HQORAZLVTCYNOFU7JAVREM5EQJSMA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.059436730 (11h54m5.80349358s) [noise=true , address=104.36.82.19:4405 , piece_id=F2TDP4KIXX7SX6JX2FOKOILDTEDBSBC2Y4XB4HO7J6NZOSSTMXCQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.067299265 (11h54m5.811356118s) [noise=true , address=104.36.82.60:4338 , piece_id=A6H2W5I3YT7NXCNCJTYZ4UFGIEDNZNRISA6RLHEMZOCUUPHYZYMQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.113365621 (11h54m5.857422471s) [noise=true , address=104.36.82.13:4189 , piece_id=RERQLVB7KU3V37Z6QVOJ5HGHV6IWXWOMJYPQMRBKLUM7ER3DUTCQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.113990013 (11h54m5.858046868s) [noise=true , address=104.36.82.15:4261 , piece_id=ZN2CAPB5FV7XPEVJARIIFNPOPEY2GOEFQFRTZZNZLPLT765MRULA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.129055509 (11h54m5.873112362s) [noise=true , address=104.36.82.4:4827 , piece_id=EGB2BTVBBNBDN57ZRD4WFTEYQ6GJCSUEKP3I6UI6XYEGFSWVQLWA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.174268649 (11h54m5.918325499s) [noise=true , address=45.91.116.11:4132 , piece_id=YL7AGZAI2AMUKWML22F64Z65AKJZR2Z3WOEUDZBNOK4IWRTSS4DQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.212907217 (11h54m5.956964063s) [noise=true , address=104.36.82.59:4303 , piece_id=HLXP2T7IJOJSVQTD6X4UJWZZ4E3HC6AXRGRTN2OUP4P3YXL4HBUQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.234310642 (11h54m5.978367495s) [noise=true , address=104.36.82.62:4417 , piece_id=JFFKIHB6YJZJLTQ5CLGATME7KA6B6TIML6XZCGYUQBFWRXAEB7DA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.270648309 (11h54m6.014705171s) [noise=true , address=104.36.82.34:4578 , piece_id=WWMNXVJFSSU3AUT5DSWNAFTTZMNFK7COL6D6P2CEH4A4O73BRDJQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.470354111 (11h54m6.214410971s) [noise=true , address=104.36.82.3:4770 , piece_id=2DHQP37D3NH2DVJTM7QANBATIVW4KP7XAXLPTJHEQN3GBQGV4F2Q, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.489350040 (11h54m6.233406907s) [noise=true , address=104.36.82.44:4892 , piece_id=W4NSH7UTQ34P5IZLD3XCALGBL6YZ5QJFVFFYVOF4DZYE4PLGC7EA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.624667783 (11h54m6.36872463s) [noise=true , address=104.36.82.45:4922 , piece_id=XKGSL3DS6JSY345IEK25AWBGSDX2IWUWDCDVXA66GBMPEO3WQWTQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.641115000 (11h54m6.385171862s) [noise=true , address=79.127.205.233:4623 , piece_id=76LKINU73EN5LDGNSBLZRTLZCWHT45L25VUP4HWRPHO2ABV6CZDA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.641149015 (11h54m6.385205876s) [seg_pos={0 0}, upload=12304]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:33:44.641367964 (11h54m6.385424829s) [noise=true , address=64.49.13.140:4506 , piece_id=Y7SSPSS7GYN7ZVI3JFV2KRPSPWKWUS44M6BPZKJCYD5NPBKZC7LA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LYkWHxg4uodPG98gJ66uz34YjsBN9N3iiy6rWQpnLjhiKoCoW): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641595473 (11h54m6.385652338s) [noise=true , address=79.127.205.234:4585 , piece_id=JZO4TRJX7CFYUKI2U7QBY54FS3NRL2GG66FDTWAKHQIDQ6ZUWSFA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GHc2AyFWF13heMfJDetaST47dXeeYuq1hB3ZjYwN7vUfXuvBF): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641716595 (11h54m6.385773458s) [noise=true , address=104.36.82.23:4462 , piece_id=CX6GMAOJFTZG2YUAQKRLCJYE2725ORXZL64IV34B2TIEPICFYXSQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641795902 (11h54m6.385852754s) [noise=true , address=104.36.82.16:4297 , piece_id=Z5ACR62PJJNYADTFWU36W6L7GZF5NYSH7KT2J66UIY63NT3U3RHA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641820947 (11h54m6.385877796s) [seg_pos={0 0}, upload=12304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:33:44.641980472 (11h54m6.386037333s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:33:44.641986313 (11h54m6.386043173s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:33:44.811637442 (11h54m6.55569429s) [upload=12304]: Done uploading segments...
Aug 7 21:33:44.811656829 (11h54m6.555713675s) [upload=12304]: Upload finished. err: <nil>
Aug 7 21:34:01.932026219 (11h54m23.676083072s) [upload=12308]: Starting upload
Aug 7 21:34:01.932128481 (11h54m23.676185355s) [upload=12308]: Done starting upload
Aug 7 21:34:01.932434039 (11h54m23.676490888s) [upload=12308]: Uploading segments...
Aug 7 21:34:01.932820553 (11h54m23.676877417s) [upload=12308]: Got next segment. Inline: false
Aug 7 21:34:01.932925753 (11h54m23.676982622s) [upload=12308]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:34:01.932934357 (11h54m23.676991224s) [upload=12308]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:34:02.022671161 (11h54m23.766728061s) [upload=12308]: Next segment err: metaclient: Too Many Requests
Aug 7 21:34:02.022701989 (11h54m23.766758862s) [upload=12308]: Done uploading segments...
Aug 7 21:34:02.022759129 (11h54m23.766815993s) [upload=12308]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.830257792s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
36 out of 60 is failed (threshold=3) [.......................XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000618293s
FAIL (2025-08-07 21:41:11 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32:34.523343267 (11h52m56.267400122s) [upload=12286]: Uploading segments... Aug 7 21:32:34.523393734 (11h52m56.267450607s) [upload=12286]: Got next segment. Inline: true Aug 7 21:32:34.523400482 (11h52m56.267457354s) [upload=12286]: Waiting for error group managing segments... Aug 7 21:32:34.523450495 (11h52m56.267507356s) [upload=12286]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:34.523455899 (11h52m56.267512759s) [upload=12286]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:34.523555922 (11h52m56.267612773s) [upload=12286]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:04.247355762 (11h53m25.991412634s) [upload=12286]: Done uploading segments... Aug 7 21:33:04.247398685 (11h53m25.991455557s) [upload=12286]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.128759856 (11h53m48.872816706s) [upload=12299]: Starting upload Aug 7 21:33:27.128845560 (11h53m48.872902422s) [upload=12299]: Done starting upload Aug 7 21:33:27.128936415 (11h53m48.872993265s) [upload=12299]: Uploading segments... Aug 7 21:33:27.128981290 (11h53m48.873038156s) [upload=12299]: Got next segment. Inline: true Aug 7 21:33:27.128986811 (11h53m48.873043661s) [upload=12299]: Waiting for error group managing segments... Aug 7 21:33:27.129031120 (11h53m48.87308798s) [upload=12299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.129036464 (11h53m48.873093314s) [upload=12299]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.129041631 (11h53m48.873098481s) [upload=12299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:56.852338652 (11h54m18.596395503s) [upload=12299]: Done uploading segments... Aug 7 21:33:56.852369847 (11h54m18.596426695s) [upload=12299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:39.735661053 (11h55m1.479717916s) [upload=12318]: Starting upload Aug 7 21:34:39.735756008 (11h55m1.479812879s) [upload=12318]: Done starting upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:41.354808211 (12h1m3.098865072s) [upload=12402]: Starting upload Aug 7 21:40:41.354938701 (12h1m3.098995568s) [upload=12402]: Done starting upload Aug 7 21:40:41.355036214 (12h1m3.099093064s) [upload=12402]: Uploading segments... Aug 7 21:40:41.355109185 (12h1m3.099166055s) [upload=12402]: Got next segment. Inline: true Aug 7 21:40:41.355116835 (12h1m3.0991737s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:40:41.355183407 (12h1m3.09924027s) [upload=12402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:40:41.355190433 (12h1m3.099247297s) [upload=12402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:41.355194357 (12h1m3.09925122s) [upload=12402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:41:11.078938147 (12h1m32.822994999s) [upload=12402]: Done uploading segments... Aug 7 21:41:11.078958711 (12h1m32.823015562s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.000659258s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
35 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001491118s
FAIL (2025-08-07 21:41:09 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001505962s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration:
1m44.411967808s
FAIL (2025-08-07 21:41:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
59m9.665872884s) [upload=12376]: Done starting upload
Aug 7 21:38:47.921973230 (11h59m9.666030093s) [upload=12376]: Uploading segments...
Aug 7 21:38:47.922447667 (11h59m9.666504541s) [upload=12376]: Got next segment. Inline: false
Aug 7 21:38:47.922523948 (11h59m9.666580816s) [upload=12376]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:38:47.922530237 (11h59m9.666587106s) [upload=12376]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:39:17.651401827 (11h59m39.395458685s) [upload=12376]: Next segment err: context deadline exceeded
Aug 7 21:39:17.651653935 (11h59m39.395710786s) [upload=12376]: Done uploading segments...
Aug 7 21:39:17.651678249 (11h59m39.395735101s) [upload=12376]: Upload finished. err: context deadline exceeded
Aug 7 21:39:20.931240120 (11h59m42.675297009s) [upload=12385]: Starting upload
Aug 7 21:39:20.931347740 (11h59m42.675404653s) [upload=12385]: Done starting upload
Aug 7 21:39:20.931824296 (11h59m42.67588115s) [upload=12385]: Uploading segments...
Aug 7 21:39:20.931983801 (11h59m42.676040662s) [upload=12385]: Got next segment. Inline: false
Aug 7 21:39:20.932060613 (11h59m42.676117474s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:39:20.932067820 (11h59m42.676124683s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:39:34.918936632 (11h59m56.662993492s) [seg_pos={0 0}, upload=12385]: Begin upload segment...
Aug 7 21:39:34.919003504 (11h59m56.663060368s) [seg_pos={0 0}, upload=12385]: Done begin upload segment.
Aug 7 21:39:34.919035204 (11h59m56.663092067s) [noise=true , address=77.90.31.18:4806 , piece_id=5VFMZTJ4UNYFVMRNXJUAAQXTOWT7GOJWM5LKVQYB4J2I6FWXY26Q, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.919666330 (11h59m56.663723198s) [noise=true , address=45.91.116.13:4153 , piece_id=VPLNW4ELDDHLPEDQ2HVWPSIBC45XN7ORVLFG75FDALY7AFYRYNEQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.920138564 (11h59m56.664195435s) [noise=true , address=104.36.82.14:4222 , piece_id=3QK6KKHJKKEFLV424LSEVYXOP63K5MTQ3YDREYFKWXFZ7ZUEO2WA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.920195139 (11h59m56.664252015s) [noise=true , address=45.91.116.8:4048 , piece_id=WWF6JJIOTPJIKOBZXVC7FRVTDSIPLRIKNN3BDJABOPOBAT7GMHZQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.920822375 (11h59m56.664879235s) [noise=true , address=104.36.82.16:4294 , piece_id=OFVD5V2JJNQF6HGZDN7P2JXRNAAT6EWL67VD4HRPGM36SWKHI2CQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.920862650 (11h59m56.664919514s) [noise=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.921441470 (11h59m56.665498332s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.921487466 (11h59m56.665544328s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.922317331 (11h59m56.666374206s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.923026433 (11h59m56.667083296s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.923407595 (11h59m56.667464455s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.923763535 (11h59m56.667820401s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.924101773 (11h59m56.668158645s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.924425635 (11h59m56.668482498s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.924925232 (11h59m56.668982102s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:34.925233497 (11h59m56.669290362s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Uploading piece...
Aug 7 21:39:35.531363219 (11h59m57.275420069s) [upload=12385]: Next returned nil segment
Aug 7 21:39:35.531379805 (11h59m57.275436664s) [upload=12385]: Waiting for error group managing segments...
Aug 7 21:39:35.709218321 (11h59m57.453275183s) [noise=true , address=104.36.82.16:4294 , piece_id=OFVD5V2JJNQF6HGZDN7P2JXRNAAT6EWL67VD4HRPGM36SWKHI2CQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.747949042 (11h59m57.492005889s) [noise=true , address=45.91.116.8:4048 , piece_id=WWF6JJIOTPJIKOBZXVC7FRVTDSIPLRIKNN3BDJABOPOBAT7GMHZQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.748418025 (11h59m57.492474871s) [noise=true , address=104.36.82.14:4222 , piece_id=3QK6KKHJKKEFLV424LSEVYXOP63K5MTQ3YDREYFKWXFZ7ZUEO2WA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.751190468 (11h59m57.495247318s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.754282840 (11h59m57.498339696s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.754668188 (11h59m57.498725035s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.760515970 (11h59m57.50457282s) [noise=true , address=45.91.116.13:4153 , piece_id=VPLNW4ELDDHLPEDQ2HVWPSIBC45XN7ORVLFG75FDALY7AFYRYNEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.765275771 (11h59m57.509332626s) [noise=true , address=77.90.31.18:4806 , piece_id=5VFMZTJ4UNYFVMRNXJUAAQXTOWT7GOJWM5LKVQYB4J2I6FWXY26Q, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.775261406 (11h59m57.519318257s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.794687407 (11h59m57.538744255s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.835050406 (11h59m57.579107253s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.835339286 (11h59m57.579396147s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.853563302 (11h59m57.597620154s) [noise=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil>
Aug 7 21:39:35.853590381 (11h59m57.597647242s) [seg_pos={0 0}, upload=12385]: Segment reached optimal threshold of 13 pieces.
Aug 7 21:39:35.853877651 (11h59m57.597934499s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Aug 7 21:39:35.854014198 (11h59m57.598071065s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Aug 7 21:39:35.854095786 (11h59m57.598152646s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection
Aug 7 21:39:35.854123369 (11h59m57.59818023s) [seg_pos={0 0}, upload=12385]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 21:39:35.854214014 (11h59m57.598270883s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:39:35.854220398 (11h59m57.598277266s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:41:05.070835032 (12h1m26.814891887s) [upload=12385]: Done uploading segments...
Aug 7 21:41:05.070862625 (12h1m26.814919475s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m44.411997688s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
4 out of 60 is failed (threshold=3) [........................................X............X.X.X..] duration:
115.759299ms remote:
"tcp:185.244.226.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:40:53 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 115.788527ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.001966062s
FAIL (2025-08-07 21:40:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001989167s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
25 out of 60 is failed (threshold=3) [..................................X.XXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.000591804s
FAIL (2025-08-07 21:40:44 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 21:30:21.398949225 (11h50m43.143006078s) [upload=12254]: Uploading segments... Aug 7 21:30:21.398993888 (11h50m43.143050757s) [upload=12254]: Got next segment. Inline: true Aug 7 21:30:21.399003181 (11h50m43.143060046s) [upload=12254]: Waiting for error group managing segments... Aug 7 21:30:21.399046484 (11h50m43.143103345s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:21.399052138 (11h50m43.143108999s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:21.399055744 (11h50m43.143112604s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:21.120945605 (11h51m42.865002833s) [upload=12254]: Done uploading segments... Aug 7 21:31:21.121002610 (11h51m42.865059457s) [upload=12254]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:21.580865540 (11h51m43.324922404s) [upload=12267]: Starting upload Aug 7 21:31:21.580962495 (11h51m43.32501936s) [upload=12267]: Done starting upload Aug 7 21:31:21.581042226 (11h51m43.325099087s) [upload=12267]: Uploading segments... Aug 7 21:31:21.581118360 (11h51m43.32517523s) [upload=12267]: Got next segment. Inline: true Aug 7 21:31:21.581124116 (11h51m43.325180981s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:31:21.581178813 (11h51m43.325235677s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:21.581184674 (11h51m43.325241538s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:21.581188187 (11h51m43.325245048s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:21.285242448 (11h52m43.029299295s) [upload=12267]: Done uploading segments... Aug 7 21:32:21.285264804 (11h52m43.029321651s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.849527306 (11h52m52.593584169s) [upload=12284]: Starting upload Aug 7 21:32:30.849593974 (11h52m52.593650837s) [upload=12284]: Done starting upload Aug 7 21:32:30.849656936 (11h52m52.593713781s) [upload=12284]: Uploading segments... Aug 7 21:32:30.849696837 (11h52m52.593753706s) [upload=12284]: Got next segment. Inline: true Aug 7 21:32:30.849701844 (11h52m52.593758708s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:32:30.849743987 (11h52m52.593800834s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.849748481 (11h52m52.59380534s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.849753457 (11h52m52.593810316s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.580274752 (11h53m52.3243316s) [upload=12284]: Done uploading segments... Aug 7 21:33:30.580297199 (11h53m52.324354049s) [upload=12284]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.996524574 (11h53m52.740581425s) [upload=12301]: Starting upload Aug 7 21:33:30.996601123 (11h53m52.740657983s) [upload=12301]: Done starting upload Aug 7 21:33:30.996679475 (11h53m52.740736329s) [upload=12301]: Uploading segments... Aug 7 21:33:30.996753641 (11h53m52.740810668s) [upload=12301]: Got next segment. Inline: true Aug 7 21:33:30.996760213 (11h53m52.740817074s) [upload=12301]: Waiting for error group managing segments... Aug 7 21:33:30.996825729 (11h53m52.740882584s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.996832674 (11h53m52.740889525s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.996838492 (11h53m52.740895343s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.718542438 (11h54m52.462599299s) [upload=12301]: Done uploading segments... Aug 7 21:34:30.718564899 (11h54m52.462621759s) [upload=12301]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.162032485 (11h54m52.906089348s) [upload=12316]: Starting upload Aug 7 21:34:31.162129373 (11h54m52.906186241s) [upload=12316]: Done starting upload Aug 7 21:34:31.162197010 (11h54m52.906253877s) [upload=12316]: Uploading segments... Aug 7 21:34:31.162265332 (11h54m52.906322201s) [upload=12316]: Got next segment. Inline: true Aug 7 21:34:31.162271192 (11h54m52.906328063s) [upload=12316]: Waiting for error group managing segments... Aug 7 21:34:31.162343122 (11h54m52.906399986s) [upload=12316]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.162350473 (11h54m52.906407351s) [upload=12316]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.162357235 (11h54m52.906414101s) [upload=12316]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.893978311 (11h55m52.638035172s) [upload=12316]: Done uploading segments... Aug 7 21:35:30.894001661 (11h55m52.638058513s) [upload=12316]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:44.593281832 (11h56m6.337338681s) [upload=12332]: Starting upload Aug 7 21:35:44.593379702 (11h56m6.337436562s) [upload=12332]: Done starting upload Aug 7 21:35:44.593440159 (11h56m6.33749701s) [upload=12332]: Uploading segments... Aug 7 21:35:44.593541679 (11h56m6.337598542s) [upload=12332]: Got next segment. Inline: true Aug 7 21:35:44.593548645 (11h56m6.337605505s) [upload=12332]: Waiting for error group managing segments... Aug 7 21:35:44.593622380 (11h56m6.337679229s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:44.593628916 (11h56m6.337685776s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:44.593632768 (11h56m6.337689686s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:44.320915363 (11h57m6.064972224s) [upload=12332]: Done uploading segments... Aug 7 21:36:44.320936819 (11h57m6.06499367s) [upload=12332]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:44.737096160 (11h57m6.48115302s) [upload=12350]: Starting upload Aug 7 21:36:44.737199233 (11h57m6.481256079s) [upload=12350]: Done starting upload Aug 7 21:36:44.737256356 (11h57m6.481313202s) [upload=12350]: Uploading segments... Aug 7 21:36:44.737309397 (11h57m6.481366267s) [upload=12350]: Got next segment. Inline: true Aug 7 21:36:44.737314124 (11h57m6.481370986s) [upload=12350]: Waiting for error group managing segments... Aug 7 21:36:44.737363580 (11h57m6.481420443s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:44.737369794 (11h57m6.481426664s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:44.737374614 (11h57m6.481431481s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:44.468181532 (11h58m6.212238379s) [upload=12350]: Done uploading segments... Aug 7 21:37:44.468202662 (11h58m6.212259509s) [upload=12350]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:44.877004541 (11h58m6.621061411s) [upload=12363]: Starting upload Aug 7 21:37:44.877093129 (11h58m6.621150048s) [upload=12363]: Done starting upload Aug 7 21:37:44.877179828 (11h58m6.621236677s) [upload=12363]: Uploading segments... Aug 7 21:37:44.877253719 (11h58m6.621310589s) [upload=12363]: Got next segment. Inline: true Aug 7 21:37:44.877262492 (11h58m6.621319365s) [upload=12363]: Waiting for error group managing segments... Aug 7 21:37:44.877313668 (11h58m6.621370528s) [upload=12363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:44.877319037 (11h58m6.621375897s) [upload=12363]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:44.877322512 (11h58m6.621379375s) [upload=12363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:44.603781017 (11h59m6.347837867s) [upload=12363]: Done uploading segments... Aug 7 21:38:44.603802220 (11h59m6.347859069s) [upload=12363]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:45.032933533 (11h59m6.776990379s) [upload=12375]: Starting upload Aug 7 21:38:45.033010670 (11h59m6.777067533s) [upload=12375]: Done starting upload Aug 7 21:38:45.033084965 (11h59m6.777141811s) [upload=12375]: Uploading segments... Aug 7 21:38:45.033156998 (11h59m6.777213868s) [upload=12375]: Got next segment. Inline: true Aug 7 21:38:45.033162279 (11h59m6.777219127s) [upload=12375]: Waiting for error group managing segments... Aug 7 21:38:45.033211860 (11h59m6.777268707s) [upload=12375]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:45.033216660 (11h59m6.77727352s) [upload=12375]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:45.033219329 (11h59m6.777276175s) [upload=12375]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:39:44.758881739 (12h0m6.502938587s) [upload=12375]: Done uploading segments... Aug 7 21:39:44.758902501 (12h0m6.502959346s) [upload=12375]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:45.185696259 (12h0m6.929753109s) [upload=12388]: Starting upload Aug 7 21:39:45.185761209 (12h0m6.929818072s) [upload=12388]: Done starting upload Aug 7 21:39:45.185834033 (12h0m6.929890896s) [upload=12388]: Uploading segments... Aug 7 21:39:45.185982126 (12h0m6.930038997s) [upload=12388]: Got next segment. Inline: true Aug 7 21:39:45.185990859 (12h0m6.930047726s) [upload=12388]: Waiting for error group managing segments... Aug 7 21:39:45.186067966 (12h0m6.930124827s) [upload=12388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:39:45.186074972 (12h0m6.930131844s) [upload=12388]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:45.186078706 (12h0m6.930135567s) [upload=12388]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:40:44.901948337 (12h1m6.646005185s) [upload=12388]: Done uploading segments... Aug 7 21:40:44.901969954 (12h1m6.646026803s) [upload=12388]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.000606261s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000592493s
FAIL (2025-08-07 21:40:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000616872s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
5 out of 60 is failed (threshold=3) [...................................XX..............XX.X.....] duration:
312.460288ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:40:36 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 312.491907ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
38 out of 60 is failed (threshold=3) [....................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000915834s
FAIL (2025-08-07 21:40:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000931089s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
36 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000601474s
FAIL (2025-08-07 21:40:27 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000623758s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
35 out of 60 is failed (threshold=3) [........................XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000759054s
FAIL (2025-08-07 21:40:26 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (11h52m6.898077597s) [upload=12272]: Uploading segments... Aug 7 21:31:45.154082030 (11h52m6.898138908s) [upload=12272]: Got next segment. Inline: true Aug 7 21:31:45.154088359 (11h52m6.898145223s) [upload=12272]: Waiting for error group managing segments... Aug 7 21:31:45.154142898 (11h52m6.898199759s) [upload=12272]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:45.154150494 (11h52m6.898207362s) [upload=12272]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:45.154155049 (11h52m6.898211914s) [upload=12272]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:14.861389353 (11h52m36.605446232s) [upload=12272]: Done uploading segments... Aug 7 21:32:14.861441235 (11h52m36.605498102s) [upload=12272]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:34.523168733 (11h52m56.267225597s) [upload=12286]: Starting upload Aug 7 21:32:34.523264231 (11h52m56.267321089s) [upload=12286]: Done starting upload Aug 7 21:32:34.523343267 (11h52m56.267400122s) [upload=12286]: Uploading segments... Aug 7 21:32:34.523393734 (11h52m56.267450607s) [upload=12286]: Got next segment. Inline: true Aug 7 21:32:34.523400482 (11h52m56.267457354s) [upload=12286]: Waiting for error group managing segments... Aug 7 21:32:34.523450495 (11h52m56.267507356s) [upload=12286]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:34.523455899 (11h52m56.267512759s) [upload=12286]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:34.523555922 (11h52m56.267612773s) [upload=12286]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:04.247355762 (11h53m25.991412634s) [upload=12286]: Done uploading segments... Aug 7 21:33:04.247398685 (11h53m25.991455557s) [upload=12286]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.128759856 (11h53m48.872816706s) [upload=12299]: Starting upload Aug 7 21:33:27.128845560 (11h53m48.872902422s) [upload=12299]: Done starting upload Aug 7 21:33:27.128936415 (11h53m48.872993265s) [upload=12299]: Uploading segments... Aug 7 21:33:27.128981290 (11h53m48.873038156s) [upload=12299]: Got next segment. Inline: true Aug 7 21:33:27.128986811 (11h53m48.873043661s) [upload=12299]: Waiting for error group managing segments... Aug 7 21:33:27.129031120 (11h53m48.87308798s) [upload=12299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.129036464 (11h53m48.873093314s) [upload=12299]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.129041631 (11h53m48.873098481s) [upload=12299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:56.852338652 (11h54m18.596395503s) [upload=12299]: Done uploading segments... Aug 7 21:33:56.852369847 (11h54m18.596426695s) [upload=12299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:39.735661053 (11h55m1.479717916s) [upload=12318]: Starting upload Aug 7 21:34:39.735756008 (11h55m1.479812879s) [upload=12318]: Done starting upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.000780951s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
28 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000602284s
FAIL (2025-08-07 21:40:25 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.000623448s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X......................X.......................XX.X........] duration:
2.984783553s
FAIL (2025-08-07 21:40:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
979s) [noise=true , address=104.36.82.44:4892 , piece_id=W4NSH7UTQ34P5IZLD3XCALGBL6YZ5QJFVFFYVOF4DZYE4PLGC7EA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304266198 (11h54m5.048323067s) [noise=true , address=104.36.82.15:4261 , piece_id=ZN2CAPB5FV7XPEVJARIIFNPOPEY2GOEFQFRTZZNZLPLT765MRULA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304638399 (11h54m5.048695263s) [noise=true , address=104.36.82.4:4827 , piece_id=EGB2BTVBBNBDN57ZRD4WFTEYQ6GJCSUEKP3I6UI6XYEGFSWVQLWA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304964313 (11h54m5.049021182s) [noise=true , address=79.127.205.234:4585 , piece_id=JZO4TRJX7CFYUKI2U7QBY54FS3NRL2GG66FDTWAKHQIDQ6ZUWSFA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.305289194 (11h54m5.049346062s) [noise=true , address=104.36.82.19:4405 , piece_id=F2TDP4KIXX7SX6JX2FOKOILDTEDBSBC2Y4XB4HO7J6NZOSSTMXCQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.305663222 (11h54m5.049720088s) [noise=true , address=104.36.82.59:4303 , piece_id=HLXP2T7IJOJSVQTD6X4UJWZZ4E3HC6AXRGRTN2OUP4P3YXL4HBUQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.307375967 (11h54m5.051432827s) [noise=true , address=64.49.13.140:4506 , piece_id=Y7SSPSS7GYN7ZVI3JFV2KRPSPWKWUS44M6BPZKJCYD5NPBKZC7LA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.307994205 (11h54m5.052051068s) [noise=true , address=104.36.82.34:4578 , piece_id=WWMNXVJFSSU3AUT5DSWNAFTTZMNFK7COL6D6P2CEH4A4O73BRDJQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.308330311 (11h54m5.052387178s) [noise=true , address=104.36.82.45:4922 , piece_id=XKGSL3DS6JSY345IEK25AWBGSDX2IWUWDCDVXA66GBMPEO3WQWTQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.308696609 (11h54m5.052753472s) [noise=true , address=104.36.82.23:4462 , piece_id=CX6GMAOJFTZG2YUAQKRLCJYE2725ORXZL64IV34B2TIEPICFYXSQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.309027383 (11h54m5.053084243s) [noise=true , address=104.36.82.16:4297 , piece_id=Z5ACR62PJJNYADTFWU36W6L7GZF5NYSH7KT2J66UIY63NT3U3RHA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.309517639 (11h54m5.053574502s) [noise=true , address=104.36.82.3:4770 , piece_id=2DHQP37D3NH2DVJTM7QANBATIVW4KP7XAXLPTJHEQN3GBQGV4F2Q, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310005964 (11h54m5.054062831s) [noise=true , address=104.36.82.37:4666 , piece_id=ONYCF5TCS7H2NQNPK53UHS5HQORAZLVTCYNOFU7JAVREM5EQJSMA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310590144 (11h54m5.05464703s) [noise=true , address=104.36.82.60:4338 , piece_id=A6H2W5I3YT7NXCNCJTYZ4UFGIEDNZNRISA6RLHEMZOCUUPHYZYMQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310999480 (11h54m5.055056348s) [noise=true , address=45.91.116.11:4132 , piece_id=YL7AGZAI2AMUKWML22F64Z65AKJZR2Z3WOEUDZBNOK4IWRTSS4DQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311301215 (11h54m5.055358078s) [noise=true , address=79.127.205.233:4623 , piece_id=76LKINU73EN5LDGNSBLZRTLZCWHT45L25VUP4HWRPHO2ABV6CZDA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311594224 (11h54m5.055651092s) [noise=true , address=104.36.82.62:4417 , piece_id=JFFKIHB6YJZJLTQ5CLGATME7KA6B6TIML6XZCGYUQBFWRXAEB7DA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311902248 (11h54m5.055959119s) [noise=true , address=104.36.82.13:4189 , piece_id=RERQLVB7KU3V37Z6QVOJ5HGHV6IWXWOMJYPQMRBKLUM7ER3DUTCQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.887542534 (11h54m5.631599386s) [upload=12304]: Next returned nil segment
Aug 7 21:33:43.887567861 (11h54m5.631624821s) [upload=12304]: Waiting for error group managing segments...
Aug 7 21:33:44.048562222 (11h54m5.792619083s) [noise=true , address=104.36.82.37:4666 , piece_id=ONYCF5TCS7H2NQNPK53UHS5HQORAZLVTCYNOFU7JAVREM5EQJSMA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.059436730 (11h54m5.80349358s) [noise=true , address=104.36.82.19:4405 , piece_id=F2TDP4KIXX7SX6JX2FOKOILDTEDBSBC2Y4XB4HO7J6NZOSSTMXCQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.067299265 (11h54m5.811356118s) [noise=true , address=104.36.82.60:4338 , piece_id=A6H2W5I3YT7NXCNCJTYZ4UFGIEDNZNRISA6RLHEMZOCUUPHYZYMQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.113365621 (11h54m5.857422471s) [noise=true , address=104.36.82.13:4189 , piece_id=RERQLVB7KU3V37Z6QVOJ5HGHV6IWXWOMJYPQMRBKLUM7ER3DUTCQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.113990013 (11h54m5.858046868s) [noise=true , address=104.36.82.15:4261 , piece_id=ZN2CAPB5FV7XPEVJARIIFNPOPEY2GOEFQFRTZZNZLPLT765MRULA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.129055509 (11h54m5.873112362s) [noise=true , address=104.36.82.4:4827 , piece_id=EGB2BTVBBNBDN57ZRD4WFTEYQ6GJCSUEKP3I6UI6XYEGFSWVQLWA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.174268649 (11h54m5.918325499s) [noise=true , address=45.91.116.11:4132 , piece_id=YL7AGZAI2AMUKWML22F64Z65AKJZR2Z3WOEUDZBNOK4IWRTSS4DQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.212907217 (11h54m5.956964063s) [noise=true , address=104.36.82.59:4303 , piece_id=HLXP2T7IJOJSVQTD6X4UJWZZ4E3HC6AXRGRTN2OUP4P3YXL4HBUQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.234310642 (11h54m5.978367495s) [noise=true , address=104.36.82.62:4417 , piece_id=JFFKIHB6YJZJLTQ5CLGATME7KA6B6TIML6XZCGYUQBFWRXAEB7DA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.270648309 (11h54m6.014705171s) [noise=true , address=104.36.82.34:4578 , piece_id=WWMNXVJFSSU3AUT5DSWNAFTTZMNFK7COL6D6P2CEH4A4O73BRDJQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.470354111 (11h54m6.214410971s) [noise=true , address=104.36.82.3:4770 , piece_id=2DHQP37D3NH2DVJTM7QANBATIVW4KP7XAXLPTJHEQN3GBQGV4F2Q, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.489350040 (11h54m6.233406907s) [noise=true , address=104.36.82.44:4892 , piece_id=W4NSH7UTQ34P5IZLD3XCALGBL6YZ5QJFVFFYVOF4DZYE4PLGC7EA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.624667783 (11h54m6.36872463s) [noise=true , address=104.36.82.45:4922 , piece_id=XKGSL3DS6JSY345IEK25AWBGSDX2IWUWDCDVXA66GBMPEO3WQWTQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.641115000 (11h54m6.385171862s) [noise=true , address=79.127.205.233:4623 , piece_id=76LKINU73EN5LDGNSBLZRTLZCWHT45L25VUP4HWRPHO2ABV6CZDA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.641149015 (11h54m6.385205876s) [seg_pos={0 0}, upload=12304]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:33:44.641367964 (11h54m6.385424829s) [noise=true , address=64.49.13.140:4506 , piece_id=Y7SSPSS7GYN7ZVI3JFV2KRPSPWKWUS44M6BPZKJCYD5NPBKZC7LA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LYkWHxg4uodPG98gJ66uz34YjsBN9N3iiy6rWQpnLjhiKoCoW): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641595473 (11h54m6.385652338s) [noise=true , address=79.127.205.234:4585 , piece_id=JZO4TRJX7CFYUKI2U7QBY54FS3NRL2GG66FDTWAKHQIDQ6ZUWSFA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GHc2AyFWF13heMfJDetaST47dXeeYuq1hB3ZjYwN7vUfXuvBF): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641716595 (11h54m6.385773458s) [noise=true , address=104.36.82.23:4462 , piece_id=CX6GMAOJFTZG2YUAQKRLCJYE2725ORXZL64IV34B2TIEPICFYXSQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641795902 (11h54m6.385852754s) [noise=true , address=104.36.82.16:4297 , piece_id=Z5ACR62PJJNYADTFWU36W6L7GZF5NYSH7KT2J66UIY63NT3U3RHA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641820947 (11h54m6.385877796s) [seg_pos={0 0}, upload=12304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:33:44.641980472 (11h54m6.386037333s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:33:44.641986313 (11h54m6.386043173s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:33:44.811637442 (11h54m6.55569429s) [upload=12304]: Done uploading segments...
Aug 7 21:33:44.811656829 (11h54m6.555713675s) [upload=12304]: Upload finished. err: <nil>
Aug 7 21:34:01.932026219 (11h54m23.676083072s) [upload=12308]: Starting upload
Aug 7 21:34:01.932128481 (11h54m23.676185355s) [upload=12308]: Done starting upload
Aug 7 21:34:01.932434039 (11h54m23.676490888s) [upload=12308]: Uploading segments...
Aug 7 21:34:01.932820553 (11h54m23.676877417s) [upload=12308]: Got next segment. Inline: false
Aug 7 21:34:01.932925753 (11h54m23.676982622s) [upload=12308]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:34:01.932934357 (11h54m23.676991224s) [upload=12308]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:34:02.022671161 (11h54m23.766728061s) [upload=12308]: Next segment err: metaclient: Too Many Requests
Aug 7 21:34:02.022701989 (11h54m23.766758862s) [upload=12308]: Done uploading segments...
Aug 7 21:34:02.022759129 (11h54m23.766815993s) [upload=12308]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.984810381s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
900.426236ms
FAIL (2025-08-07 21:40:08 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests
Duration: 900.473265ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
4 out of 60 is failed (threshold=3) [.........................................X............X.X.X.] duration:
287.795373ms remote:
"tcp:185.244.226.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:40:02 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 287.818533ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
37 out of 60 is failed (threshold=3) [.....................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001125472s
FAIL (2025-08-07 21:40:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001150953s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
4 out of 60 is failed (threshold=3) [..........................................X............X.X.X] duration:
2m0.00691948s remote:
"tcp:185.244.226.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:40:01 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 2m0.006948651s
edge
gateway
vhost
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
5 out of 60 is failed (threshold=3) [....................................XX..............XX.X....] duration:
418.631245ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:40:00 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 418.654955ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X......................X.......................XX.X.......] duration:
3.26763924s
FAIL (2025-08-07 21:39:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
979s) [noise=true , address=104.36.82.44:4892 , piece_id=W4NSH7UTQ34P5IZLD3XCALGBL6YZ5QJFVFFYVOF4DZYE4PLGC7EA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304266198 (11h54m5.048323067s) [noise=true , address=104.36.82.15:4261 , piece_id=ZN2CAPB5FV7XPEVJARIIFNPOPEY2GOEFQFRTZZNZLPLT765MRULA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304638399 (11h54m5.048695263s) [noise=true , address=104.36.82.4:4827 , piece_id=EGB2BTVBBNBDN57ZRD4WFTEYQ6GJCSUEKP3I6UI6XYEGFSWVQLWA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304964313 (11h54m5.049021182s) [noise=true , address=79.127.205.234:4585 , piece_id=JZO4TRJX7CFYUKI2U7QBY54FS3NRL2GG66FDTWAKHQIDQ6ZUWSFA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.305289194 (11h54m5.049346062s) [noise=true , address=104.36.82.19:4405 , piece_id=F2TDP4KIXX7SX6JX2FOKOILDTEDBSBC2Y4XB4HO7J6NZOSSTMXCQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.305663222 (11h54m5.049720088s) [noise=true , address=104.36.82.59:4303 , piece_id=HLXP2T7IJOJSVQTD6X4UJWZZ4E3HC6AXRGRTN2OUP4P3YXL4HBUQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.307375967 (11h54m5.051432827s) [noise=true , address=64.49.13.140:4506 , piece_id=Y7SSPSS7GYN7ZVI3JFV2KRPSPWKWUS44M6BPZKJCYD5NPBKZC7LA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.307994205 (11h54m5.052051068s) [noise=true , address=104.36.82.34:4578 , piece_id=WWMNXVJFSSU3AUT5DSWNAFTTZMNFK7COL6D6P2CEH4A4O73BRDJQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.308330311 (11h54m5.052387178s) [noise=true , address=104.36.82.45:4922 , piece_id=XKGSL3DS6JSY345IEK25AWBGSDX2IWUWDCDVXA66GBMPEO3WQWTQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.308696609 (11h54m5.052753472s) [noise=true , address=104.36.82.23:4462 , piece_id=CX6GMAOJFTZG2YUAQKRLCJYE2725ORXZL64IV34B2TIEPICFYXSQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.309027383 (11h54m5.053084243s) [noise=true , address=104.36.82.16:4297 , piece_id=Z5ACR62PJJNYADTFWU36W6L7GZF5NYSH7KT2J66UIY63NT3U3RHA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.309517639 (11h54m5.053574502s) [noise=true , address=104.36.82.3:4770 , piece_id=2DHQP37D3NH2DVJTM7QANBATIVW4KP7XAXLPTJHEQN3GBQGV4F2Q, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310005964 (11h54m5.054062831s) [noise=true , address=104.36.82.37:4666 , piece_id=ONYCF5TCS7H2NQNPK53UHS5HQORAZLVTCYNOFU7JAVREM5EQJSMA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310590144 (11h54m5.05464703s) [noise=true , address=104.36.82.60:4338 , piece_id=A6H2W5I3YT7NXCNCJTYZ4UFGIEDNZNRISA6RLHEMZOCUUPHYZYMQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310999480 (11h54m5.055056348s) [noise=true , address=45.91.116.11:4132 , piece_id=YL7AGZAI2AMUKWML22F64Z65AKJZR2Z3WOEUDZBNOK4IWRTSS4DQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311301215 (11h54m5.055358078s) [noise=true , address=79.127.205.233:4623 , piece_id=76LKINU73EN5LDGNSBLZRTLZCWHT45L25VUP4HWRPHO2ABV6CZDA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311594224 (11h54m5.055651092s) [noise=true , address=104.36.82.62:4417 , piece_id=JFFKIHB6YJZJLTQ5CLGATME7KA6B6TIML6XZCGYUQBFWRXAEB7DA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311902248 (11h54m5.055959119s) [noise=true , address=104.36.82.13:4189 , piece_id=RERQLVB7KU3V37Z6QVOJ5HGHV6IWXWOMJYPQMRBKLUM7ER3DUTCQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.887542534 (11h54m5.631599386s) [upload=12304]: Next returned nil segment
Aug 7 21:33:43.887567861 (11h54m5.631624821s) [upload=12304]: Waiting for error group managing segments...
Aug 7 21:33:44.048562222 (11h54m5.792619083s) [noise=true , address=104.36.82.37:4666 , piece_id=ONYCF5TCS7H2NQNPK53UHS5HQORAZLVTCYNOFU7JAVREM5EQJSMA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.059436730 (11h54m5.80349358s) [noise=true , address=104.36.82.19:4405 , piece_id=F2TDP4KIXX7SX6JX2FOKOILDTEDBSBC2Y4XB4HO7J6NZOSSTMXCQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.067299265 (11h54m5.811356118s) [noise=true , address=104.36.82.60:4338 , piece_id=A6H2W5I3YT7NXCNCJTYZ4UFGIEDNZNRISA6RLHEMZOCUUPHYZYMQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.113365621 (11h54m5.857422471s) [noise=true , address=104.36.82.13:4189 , piece_id=RERQLVB7KU3V37Z6QVOJ5HGHV6IWXWOMJYPQMRBKLUM7ER3DUTCQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.113990013 (11h54m5.858046868s) [noise=true , address=104.36.82.15:4261 , piece_id=ZN2CAPB5FV7XPEVJARIIFNPOPEY2GOEFQFRTZZNZLPLT765MRULA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.129055509 (11h54m5.873112362s) [noise=true , address=104.36.82.4:4827 , piece_id=EGB2BTVBBNBDN57ZRD4WFTEYQ6GJCSUEKP3I6UI6XYEGFSWVQLWA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.174268649 (11h54m5.918325499s) [noise=true , address=45.91.116.11:4132 , piece_id=YL7AGZAI2AMUKWML22F64Z65AKJZR2Z3WOEUDZBNOK4IWRTSS4DQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.212907217 (11h54m5.956964063s) [noise=true , address=104.36.82.59:4303 , piece_id=HLXP2T7IJOJSVQTD6X4UJWZZ4E3HC6AXRGRTN2OUP4P3YXL4HBUQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.234310642 (11h54m5.978367495s) [noise=true , address=104.36.82.62:4417 , piece_id=JFFKIHB6YJZJLTQ5CLGATME7KA6B6TIML6XZCGYUQBFWRXAEB7DA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.270648309 (11h54m6.014705171s) [noise=true , address=104.36.82.34:4578 , piece_id=WWMNXVJFSSU3AUT5DSWNAFTTZMNFK7COL6D6P2CEH4A4O73BRDJQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.470354111 (11h54m6.214410971s) [noise=true , address=104.36.82.3:4770 , piece_id=2DHQP37D3NH2DVJTM7QANBATIVW4KP7XAXLPTJHEQN3GBQGV4F2Q, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.489350040 (11h54m6.233406907s) [noise=true , address=104.36.82.44:4892 , piece_id=W4NSH7UTQ34P5IZLD3XCALGBL6YZ5QJFVFFYVOF4DZYE4PLGC7EA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.624667783 (11h54m6.36872463s) [noise=true , address=104.36.82.45:4922 , piece_id=XKGSL3DS6JSY345IEK25AWBGSDX2IWUWDCDVXA66GBMPEO3WQWTQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.641115000 (11h54m6.385171862s) [noise=true , address=79.127.205.233:4623 , piece_id=76LKINU73EN5LDGNSBLZRTLZCWHT45L25VUP4HWRPHO2ABV6CZDA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.641149015 (11h54m6.385205876s) [seg_pos={0 0}, upload=12304]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:33:44.641367964 (11h54m6.385424829s) [noise=true , address=64.49.13.140:4506 , piece_id=Y7SSPSS7GYN7ZVI3JFV2KRPSPWKWUS44M6BPZKJCYD5NPBKZC7LA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LYkWHxg4uodPG98gJ66uz34YjsBN9N3iiy6rWQpnLjhiKoCoW): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641595473 (11h54m6.385652338s) [noise=true , address=79.127.205.234:4585 , piece_id=JZO4TRJX7CFYUKI2U7QBY54FS3NRL2GG66FDTWAKHQIDQ6ZUWSFA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GHc2AyFWF13heMfJDetaST47dXeeYuq1hB3ZjYwN7vUfXuvBF): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641716595 (11h54m6.385773458s) [noise=true , address=104.36.82.23:4462 , piece_id=CX6GMAOJFTZG2YUAQKRLCJYE2725ORXZL64IV34B2TIEPICFYXSQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641795902 (11h54m6.385852754s) [noise=true , address=104.36.82.16:4297 , piece_id=Z5ACR62PJJNYADTFWU36W6L7GZF5NYSH7KT2J66UIY63NT3U3RHA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641820947 (11h54m6.385877796s) [seg_pos={0 0}, upload=12304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:33:44.641980472 (11h54m6.386037333s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:33:44.641986313 (11h54m6.386043173s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:33:44.811637442 (11h54m6.55569429s) [upload=12304]: Done uploading segments...
Aug 7 21:33:44.811656829 (11h54m6.555713675s) [upload=12304]: Upload finished. err: <nil>
Aug 7 21:34:01.932026219 (11h54m23.676083072s) [upload=12308]: Starting upload
Aug 7 21:34:01.932128481 (11h54m23.676185355s) [upload=12308]: Done starting upload
Aug 7 21:34:01.932434039 (11h54m23.676490888s) [upload=12308]: Uploading segments...
Aug 7 21:34:01.932820553 (11h54m23.676877417s) [upload=12308]: Got next segment. Inline: false
Aug 7 21:34:01.932925753 (11h54m23.676982622s) [upload=12308]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:34:01.932934357 (11h54m23.676991224s) [upload=12308]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:34:02.022671161 (11h54m23.766728061s) [upload=12308]: Next segment err: metaclient: Too Many Requests
Aug 7 21:34:02.022701989 (11h54m23.766758862s) [upload=12308]: Done uploading segments...
Aug 7 21:34:02.022759129 (11h54m23.766815993s) [upload=12308]: Upload finished. err: metaclient: Too Many Requests
Duration: 3.26766701s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.001158298s
FAIL (2025-08-07 21:39:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001184731s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001259839s
FAIL (2025-08-07 21:39:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00127737s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
24 out of 60 is failed (threshold=3) [...................................X.XXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.000799411s
FAIL (2025-08-07 21:39:44 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 21:29:21.230482429 (11h49m42.974539278s) [upload=12238]: Uploading segments... Aug 7 21:29:21.230606006 (11h49m42.974662867s) [upload=12238]: Got next segment. Inline: true Aug 7 21:29:21.230613609 (11h49m42.974670469s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:29:21.230660759 (11h49m42.974717609s) [upload=12238]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:21.230666510 (11h49m42.974723374s) [upload=12238]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:21.230691185 (11h49m42.974748046s) [upload=12238]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:20.959347392 (11h50m42.703404243s) [upload=12238]: Done uploading segments... Aug 7 21:30:20.959370210 (11h50m42.70342706s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:21.398814384 (11h50m43.142871245s) [upload=12254]: Starting upload Aug 7 21:30:21.398880649 (11h50m43.142937512s) [upload=12254]: Done starting upload Aug 7 21:30:21.398949225 (11h50m43.143006078s) [upload=12254]: Uploading segments... Aug 7 21:30:21.398993888 (11h50m43.143050757s) [upload=12254]: Got next segment. Inline: true Aug 7 21:30:21.399003181 (11h50m43.143060046s) [upload=12254]: Waiting for error group managing segments... Aug 7 21:30:21.399046484 (11h50m43.143103345s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:21.399052138 (11h50m43.143108999s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:21.399055744 (11h50m43.143112604s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:21.120945605 (11h51m42.865002833s) [upload=12254]: Done uploading segments... Aug 7 21:31:21.121002610 (11h51m42.865059457s) [upload=12254]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:21.580865540 (11h51m43.324922404s) [upload=12267]: Starting upload Aug 7 21:31:21.580962495 (11h51m43.32501936s) [upload=12267]: Done starting upload Aug 7 21:31:21.581042226 (11h51m43.325099087s) [upload=12267]: Uploading segments... Aug 7 21:31:21.581118360 (11h51m43.32517523s) [upload=12267]: Got next segment. Inline: true Aug 7 21:31:21.581124116 (11h51m43.325180981s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:31:21.581178813 (11h51m43.325235677s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:21.581184674 (11h51m43.325241538s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:21.581188187 (11h51m43.325245048s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:21.285242448 (11h52m43.029299295s) [upload=12267]: Done uploading segments... Aug 7 21:32:21.285264804 (11h52m43.029321651s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.849527306 (11h52m52.593584169s) [upload=12284]: Starting upload Aug 7 21:32:30.849593974 (11h52m52.593650837s) [upload=12284]: Done starting upload Aug 7 21:32:30.849656936 (11h52m52.593713781s) [upload=12284]: Uploading segments... Aug 7 21:32:30.849696837 (11h52m52.593753706s) [upload=12284]: Got next segment. Inline: true Aug 7 21:32:30.849701844 (11h52m52.593758708s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:32:30.849743987 (11h52m52.593800834s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.849748481 (11h52m52.59380534s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.849753457 (11h52m52.593810316s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.580274752 (11h53m52.3243316s) [upload=12284]: Done uploading segments... Aug 7 21:33:30.580297199 (11h53m52.324354049s) [upload=12284]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.996524574 (11h53m52.740581425s) [upload=12301]: Starting upload Aug 7 21:33:30.996601123 (11h53m52.740657983s) [upload=12301]: Done starting upload Aug 7 21:33:30.996679475 (11h53m52.740736329s) [upload=12301]: Uploading segments... Aug 7 21:33:30.996753641 (11h53m52.740810668s) [upload=12301]: Got next segment. Inline: true Aug 7 21:33:30.996760213 (11h53m52.740817074s) [upload=12301]: Waiting for error group managing segments... Aug 7 21:33:30.996825729 (11h53m52.740882584s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.996832674 (11h53m52.740889525s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.996838492 (11h53m52.740895343s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.718542438 (11h54m52.462599299s) [upload=12301]: Done uploading segments... Aug 7 21:34:30.718564899 (11h54m52.462621759s) [upload=12301]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.162032485 (11h54m52.906089348s) [upload=12316]: Starting upload Aug 7 21:34:31.162129373 (11h54m52.906186241s) [upload=12316]: Done starting upload Aug 7 21:34:31.162197010 (11h54m52.906253877s) [upload=12316]: Uploading segments... Aug 7 21:34:31.162265332 (11h54m52.906322201s) [upload=12316]: Got next segment. Inline: true Aug 7 21:34:31.162271192 (11h54m52.906328063s) [upload=12316]: Waiting for error group managing segments... Aug 7 21:34:31.162343122 (11h54m52.906399986s) [upload=12316]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.162350473 (11h54m52.906407351s) [upload=12316]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.162357235 (11h54m52.906414101s) [upload=12316]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.893978311 (11h55m52.638035172s) [upload=12316]: Done uploading segments... Aug 7 21:35:30.894001661 (11h55m52.638058513s) [upload=12316]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:44.593281832 (11h56m6.337338681s) [upload=12332]: Starting upload Aug 7 21:35:44.593379702 (11h56m6.337436562s) [upload=12332]: Done starting upload Aug 7 21:35:44.593440159 (11h56m6.33749701s) [upload=12332]: Uploading segments... Aug 7 21:35:44.593541679 (11h56m6.337598542s) [upload=12332]: Got next segment. Inline: true Aug 7 21:35:44.593548645 (11h56m6.337605505s) [upload=12332]: Waiting for error group managing segments... Aug 7 21:35:44.593622380 (11h56m6.337679229s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:44.593628916 (11h56m6.337685776s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:44.593632768 (11h56m6.337689686s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:44.320915363 (11h57m6.064972224s) [upload=12332]: Done uploading segments... Aug 7 21:36:44.320936819 (11h57m6.06499367s) [upload=12332]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:44.737096160 (11h57m6.48115302s) [upload=12350]: Starting upload Aug 7 21:36:44.737199233 (11h57m6.481256079s) [upload=12350]: Done starting upload Aug 7 21:36:44.737256356 (11h57m6.481313202s) [upload=12350]: Uploading segments... Aug 7 21:36:44.737309397 (11h57m6.481366267s) [upload=12350]: Got next segment. Inline: true Aug 7 21:36:44.737314124 (11h57m6.481370986s) [upload=12350]: Waiting for error group managing segments... Aug 7 21:36:44.737363580 (11h57m6.481420443s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:44.737369794 (11h57m6.481426664s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:44.737374614 (11h57m6.481431481s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:44.468181532 (11h58m6.212238379s) [upload=12350]: Done uploading segments... Aug 7 21:37:44.468202662 (11h58m6.212259509s) [upload=12350]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:44.877004541 (11h58m6.621061411s) [upload=12363]: Starting upload Aug 7 21:37:44.877093129 (11h58m6.621150048s) [upload=12363]: Done starting upload Aug 7 21:37:44.877179828 (11h58m6.621236677s) [upload=12363]: Uploading segments... Aug 7 21:37:44.877253719 (11h58m6.621310589s) [upload=12363]: Got next segment. Inline: true Aug 7 21:37:44.877262492 (11h58m6.621319365s) [upload=12363]: Waiting for error group managing segments... Aug 7 21:37:44.877313668 (11h58m6.621370528s) [upload=12363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:44.877319037 (11h58m6.621375897s) [upload=12363]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:44.877322512 (11h58m6.621379375s) [upload=12363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:44.603781017 (11h59m6.347837867s) [upload=12363]: Done uploading segments... Aug 7 21:38:44.603802220 (11h59m6.347859069s) [upload=12363]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:45.032933533 (11h59m6.776990379s) [upload=12375]: Starting upload Aug 7 21:38:45.033010670 (11h59m6.777067533s) [upload=12375]: Done starting upload Aug 7 21:38:45.033084965 (11h59m6.777141811s) [upload=12375]: Uploading segments... Aug 7 21:38:45.033156998 (11h59m6.777213868s) [upload=12375]: Got next segment. Inline: true Aug 7 21:38:45.033162279 (11h59m6.777219127s) [upload=12375]: Waiting for error group managing segments... Aug 7 21:38:45.033211860 (11h59m6.777268707s) [upload=12375]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:45.033216660 (11h59m6.77727352s) [upload=12375]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:45.033219329 (11h59m6.777276175s) [upload=12375]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:39:44.758881739 (12h0m6.502938587s) [upload=12375]: Done uploading segments... Aug 7 21:39:44.758902501 (12h0m6.502959346s) [upload=12375]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.000829447s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X......................X.......................XX.X......] duration:
3.061000891s
FAIL (2025-08-07 21:39:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
979s) [noise=true , address=104.36.82.44:4892 , piece_id=W4NSH7UTQ34P5IZLD3XCALGBL6YZ5QJFVFFYVOF4DZYE4PLGC7EA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304266198 (11h54m5.048323067s) [noise=true , address=104.36.82.15:4261 , piece_id=ZN2CAPB5FV7XPEVJARIIFNPOPEY2GOEFQFRTZZNZLPLT765MRULA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304638399 (11h54m5.048695263s) [noise=true , address=104.36.82.4:4827 , piece_id=EGB2BTVBBNBDN57ZRD4WFTEYQ6GJCSUEKP3I6UI6XYEGFSWVQLWA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.304964313 (11h54m5.049021182s) [noise=true , address=79.127.205.234:4585 , piece_id=JZO4TRJX7CFYUKI2U7QBY54FS3NRL2GG66FDTWAKHQIDQ6ZUWSFA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.305289194 (11h54m5.049346062s) [noise=true , address=104.36.82.19:4405 , piece_id=F2TDP4KIXX7SX6JX2FOKOILDTEDBSBC2Y4XB4HO7J6NZOSSTMXCQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.305663222 (11h54m5.049720088s) [noise=true , address=104.36.82.59:4303 , piece_id=HLXP2T7IJOJSVQTD6X4UJWZZ4E3HC6AXRGRTN2OUP4P3YXL4HBUQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.307375967 (11h54m5.051432827s) [noise=true , address=64.49.13.140:4506 , piece_id=Y7SSPSS7GYN7ZVI3JFV2KRPSPWKWUS44M6BPZKJCYD5NPBKZC7LA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.307994205 (11h54m5.052051068s) [noise=true , address=104.36.82.34:4578 , piece_id=WWMNXVJFSSU3AUT5DSWNAFTTZMNFK7COL6D6P2CEH4A4O73BRDJQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.308330311 (11h54m5.052387178s) [noise=true , address=104.36.82.45:4922 , piece_id=XKGSL3DS6JSY345IEK25AWBGSDX2IWUWDCDVXA66GBMPEO3WQWTQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.308696609 (11h54m5.052753472s) [noise=true , address=104.36.82.23:4462 , piece_id=CX6GMAOJFTZG2YUAQKRLCJYE2725ORXZL64IV34B2TIEPICFYXSQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.309027383 (11h54m5.053084243s) [noise=true , address=104.36.82.16:4297 , piece_id=Z5ACR62PJJNYADTFWU36W6L7GZF5NYSH7KT2J66UIY63NT3U3RHA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.309517639 (11h54m5.053574502s) [noise=true , address=104.36.82.3:4770 , piece_id=2DHQP37D3NH2DVJTM7QANBATIVW4KP7XAXLPTJHEQN3GBQGV4F2Q, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310005964 (11h54m5.054062831s) [noise=true , address=104.36.82.37:4666 , piece_id=ONYCF5TCS7H2NQNPK53UHS5HQORAZLVTCYNOFU7JAVREM5EQJSMA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310590144 (11h54m5.05464703s) [noise=true , address=104.36.82.60:4338 , piece_id=A6H2W5I3YT7NXCNCJTYZ4UFGIEDNZNRISA6RLHEMZOCUUPHYZYMQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.310999480 (11h54m5.055056348s) [noise=true , address=45.91.116.11:4132 , piece_id=YL7AGZAI2AMUKWML22F64Z65AKJZR2Z3WOEUDZBNOK4IWRTSS4DQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311301215 (11h54m5.055358078s) [noise=true , address=79.127.205.233:4623 , piece_id=76LKINU73EN5LDGNSBLZRTLZCWHT45L25VUP4HWRPHO2ABV6CZDA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311594224 (11h54m5.055651092s) [noise=true , address=104.36.82.62:4417 , piece_id=JFFKIHB6YJZJLTQ5CLGATME7KA6B6TIML6XZCGYUQBFWRXAEB7DA, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.311902248 (11h54m5.055959119s) [noise=true , address=104.36.82.13:4189 , piece_id=RERQLVB7KU3V37Z6QVOJ5HGHV6IWXWOMJYPQMRBKLUM7ER3DUTCQ, seg_pos={0 0}, upload=12304]: Uploading piece...
Aug 7 21:33:43.887542534 (11h54m5.631599386s) [upload=12304]: Next returned nil segment
Aug 7 21:33:43.887567861 (11h54m5.631624821s) [upload=12304]: Waiting for error group managing segments...
Aug 7 21:33:44.048562222 (11h54m5.792619083s) [noise=true , address=104.36.82.37:4666 , piece_id=ONYCF5TCS7H2NQNPK53UHS5HQORAZLVTCYNOFU7JAVREM5EQJSMA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.059436730 (11h54m5.80349358s) [noise=true , address=104.36.82.19:4405 , piece_id=F2TDP4KIXX7SX6JX2FOKOILDTEDBSBC2Y4XB4HO7J6NZOSSTMXCQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.067299265 (11h54m5.811356118s) [noise=true , address=104.36.82.60:4338 , piece_id=A6H2W5I3YT7NXCNCJTYZ4UFGIEDNZNRISA6RLHEMZOCUUPHYZYMQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.113365621 (11h54m5.857422471s) [noise=true , address=104.36.82.13:4189 , piece_id=RERQLVB7KU3V37Z6QVOJ5HGHV6IWXWOMJYPQMRBKLUM7ER3DUTCQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.113990013 (11h54m5.858046868s) [noise=true , address=104.36.82.15:4261 , piece_id=ZN2CAPB5FV7XPEVJARIIFNPOPEY2GOEFQFRTZZNZLPLT765MRULA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.129055509 (11h54m5.873112362s) [noise=true , address=104.36.82.4:4827 , piece_id=EGB2BTVBBNBDN57ZRD4WFTEYQ6GJCSUEKP3I6UI6XYEGFSWVQLWA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.174268649 (11h54m5.918325499s) [noise=true , address=45.91.116.11:4132 , piece_id=YL7AGZAI2AMUKWML22F64Z65AKJZR2Z3WOEUDZBNOK4IWRTSS4DQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.212907217 (11h54m5.956964063s) [noise=true , address=104.36.82.59:4303 , piece_id=HLXP2T7IJOJSVQTD6X4UJWZZ4E3HC6AXRGRTN2OUP4P3YXL4HBUQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.234310642 (11h54m5.978367495s) [noise=true , address=104.36.82.62:4417 , piece_id=JFFKIHB6YJZJLTQ5CLGATME7KA6B6TIML6XZCGYUQBFWRXAEB7DA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.270648309 (11h54m6.014705171s) [noise=true , address=104.36.82.34:4578 , piece_id=WWMNXVJFSSU3AUT5DSWNAFTTZMNFK7COL6D6P2CEH4A4O73BRDJQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.470354111 (11h54m6.214410971s) [noise=true , address=104.36.82.3:4770 , piece_id=2DHQP37D3NH2DVJTM7QANBATIVW4KP7XAXLPTJHEQN3GBQGV4F2Q, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.489350040 (11h54m6.233406907s) [noise=true , address=104.36.82.44:4892 , piece_id=W4NSH7UTQ34P5IZLD3XCALGBL6YZ5QJFVFFYVOF4DZYE4PLGC7EA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.624667783 (11h54m6.36872463s) [noise=true , address=104.36.82.45:4922 , piece_id=XKGSL3DS6JSY345IEK25AWBGSDX2IWUWDCDVXA66GBMPEO3WQWTQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.641115000 (11h54m6.385171862s) [noise=true , address=79.127.205.233:4623 , piece_id=76LKINU73EN5LDGNSBLZRTLZCWHT45L25VUP4HWRPHO2ABV6CZDA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil>
Aug 7 21:33:44.641149015 (11h54m6.385205876s) [seg_pos={0 0}, upload=12304]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:33:44.641367964 (11h54m6.385424829s) [noise=true , address=64.49.13.140:4506 , piece_id=Y7SSPSS7GYN7ZVI3JFV2KRPSPWKWUS44M6BPZKJCYD5NPBKZC7LA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LYkWHxg4uodPG98gJ66uz34YjsBN9N3iiy6rWQpnLjhiKoCoW): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641595473 (11h54m6.385652338s) [noise=true , address=79.127.205.234:4585 , piece_id=JZO4TRJX7CFYUKI2U7QBY54FS3NRL2GG66FDTWAKHQIDQ6ZUWSFA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GHc2AyFWF13heMfJDetaST47dXeeYuq1hB3ZjYwN7vUfXuvBF): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641716595 (11h54m6.385773458s) [noise=true , address=104.36.82.23:4462 , piece_id=CX6GMAOJFTZG2YUAQKRLCJYE2725ORXZL64IV34B2TIEPICFYXSQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641795902 (11h54m6.385852754s) [noise=true , address=104.36.82.16:4297 , piece_id=Z5ACR62PJJNYADTFWU36W6L7GZF5NYSH7KT2J66UIY63NT3U3RHA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF
Aug 7 21:33:44.641820947 (11h54m6.385877796s) [seg_pos={0 0}, upload=12304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:33:44.641980472 (11h54m6.386037333s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:33:44.641986313 (11h54m6.386043173s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:33:44.811637442 (11h54m6.55569429s) [upload=12304]: Done uploading segments...
Aug 7 21:33:44.811656829 (11h54m6.555713675s) [upload=12304]: Upload finished. err: <nil>
Aug 7 21:34:01.932026219 (11h54m23.676083072s) [upload=12308]: Starting upload
Aug 7 21:34:01.932128481 (11h54m23.676185355s) [upload=12308]: Done starting upload
Aug 7 21:34:01.932434039 (11h54m23.676490888s) [upload=12308]: Uploading segments...
Aug 7 21:34:01.932820553 (11h54m23.676877417s) [upload=12308]: Got next segment. Inline: false
Aug 7 21:34:01.932925753 (11h54m23.676982622s) [upload=12308]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:34:01.932934357 (11h54m23.676991224s) [upload=12308]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:34:02.022671161 (11h54m23.766728061s) [upload=12308]: Next segment err: metaclient: Too Many Requests
Aug 7 21:34:02.022701989 (11h54m23.766758862s) [upload=12308]: Done uploading segments...
Aug 7 21:34:02.022759129 (11h54m23.766815993s) [upload=12308]: Upload finished. err: metaclient: Too Many Requests
Duration: 3.061032263s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
35 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001274897s
FAIL (2025-08-07 21:39:32 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.00129631s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
34 out of 60 is failed (threshold=3) [.........................XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000535955s
FAIL (2025-08-07 21:39:27 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 51m32.903767956s) [upload=12264]: Uploading segments... Aug 7 21:31:11.159755148 (11h51m32.903812017s) [upload=12264]: Got next segment. Inline: true Aug 7 21:31:11.159760622 (11h51m32.903817485s) [upload=12264]: Waiting for error group managing segments... Aug 7 21:31:11.159814091 (11h51m32.903870951s) [upload=12264]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:11.159819842 (11h51m32.903876703s) [upload=12264]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:11.159823678 (11h51m32.903880539s) [upload=12264]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:40.887250927 (11h52m2.631307774s) [upload=12264]: Done uploading segments... Aug 7 21:31:40.887302452 (11h52m2.631359314s) [upload=12264]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:45.153828960 (11h52m6.89788582s) [upload=12272]: Starting upload Aug 7 21:31:45.153940931 (11h52m6.897997802s) [upload=12272]: Done starting upload Aug 7 21:31:45.154020734 (11h52m6.898077597s) [upload=12272]: Uploading segments... Aug 7 21:31:45.154082030 (11h52m6.898138908s) [upload=12272]: Got next segment. Inline: true Aug 7 21:31:45.154088359 (11h52m6.898145223s) [upload=12272]: Waiting for error group managing segments... Aug 7 21:31:45.154142898 (11h52m6.898199759s) [upload=12272]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:45.154150494 (11h52m6.898207362s) [upload=12272]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:45.154155049 (11h52m6.898211914s) [upload=12272]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:14.861389353 (11h52m36.605446232s) [upload=12272]: Done uploading segments... Aug 7 21:32:14.861441235 (11h52m36.605498102s) [upload=12272]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:34.523168733 (11h52m56.267225597s) [upload=12286]: Starting upload Aug 7 21:32:34.523264231 (11h52m56.267321089s) [upload=12286]: Done starting upload Aug 7 21:32:34.523343267 (11h52m56.267400122s) [upload=12286]: Uploading segments... Aug 7 21:32:34.523393734 (11h52m56.267450607s) [upload=12286]: Got next segment. Inline: true Aug 7 21:32:34.523400482 (11h52m56.267457354s) [upload=12286]: Waiting for error group managing segments... Aug 7 21:32:34.523450495 (11h52m56.267507356s) [upload=12286]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:34.523455899 (11h52m56.267512759s) [upload=12286]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:34.523555922 (11h52m56.267612773s) [upload=12286]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:04.247355762 (11h53m25.991412634s) [upload=12286]: Done uploading segments... Aug 7 21:33:04.247398685 (11h53m25.991455557s) [upload=12286]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.128759856 (11h53m48.872816706s) [upload=12299]: Starting upload Aug 7 21:33:27.128845560 (11h53m48.872902422s) [upload=12299]: Done starting upload Aug 7 21:33:27.128936415 (11h53m48.872993265s) [upload=12299]: Uploading segments... Aug 7 21:33:27.128981290 (11h53m48.873038156s) [upload=12299]: Got next segment. Inline: true Aug 7 21:33:27.128986811 (11h53m48.873043661s) [upload=12299]: Waiting for error group managing segments... Aug 7 21:33:27.129031120 (11h53m48.87308798s) [upload=12299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.129036464 (11h53m48.873093314s) [upload=12299]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.129041631 (11h53m48.873098481s) [upload=12299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:56.852338652 (11h54m18.596395503s) [upload=12299]: Done uploading segments... Aug 7 21:33:56.852369847 (11h54m18.596426695s) [upload=12299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:39.735661053 (11h55m1.479717916s) [upload=12318]: Starting upload Aug 7 21:34:39.735756008 (11h55m1.479812879s) [upload=12318]: Done starting upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.000548817s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
36 out of 60 is failed (threshold=3) [......................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000959525s
FAIL (2025-08-07 21:39:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000979402s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration:
30.001640709s
FAIL (2025-08-07 21:39:17 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== hRequestItem_ObjectCommit Aug 7 21:31:24.464709964 (11h51m46.208766834s) [upload=12096]: Done uploading segments... Aug 7 21:31:24.464745783 (11h51m46.208802653s) [upload=12096]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:24.881904902 (11h51m46.625961766s) [upload=12268]: Starting upload Aug 7 21:31:24.882094151 (11h51m46.626151016s) [upload=12268]: Done starting upload Aug 7 21:31:24.882164223 (11h51m46.626221082s) [upload=12268]: Uploading segments... Aug 7 21:31:24.882475408 (11h51m46.626532277s) [upload=12268]: Got next segment. Inline: false Aug 7 21:31:24.882552261 (11h51m46.626609127s) [upload=12268]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:24.882560165 (11h51m46.626617032s) [upload=12268]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:54.614102730 (11h52m16.358159588s) [upload=12268]: Next segment err: context deadline exceeded Aug 7 21:31:54.614205863 (11h52m16.358262714s) [upload=12268]: Done uploading segments... Aug 7 21:31:54.614225924 (11h52m16.358282773s) [upload=12268]: Upload finished. err: context deadline exceeded Aug 7 21:32:00.253568814 (11h52m21.997625667s) [upload=12278]: Starting upload Aug 7 21:32:00.253669214 (11h52m21.997726066s) [upload=12278]: Done starting upload Aug 7 21:32:00.254008956 (11h52m21.998065808s) [upload=12278]: Uploading segments... Aug 7 21:32:00.254068254 (11h52m21.998125125s) [upload=12278]: Got next segment. Inline: false Aug 7 21:32:00.254112779 (11h52m21.998169639s) [upload=12278]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:00.254118157 (11h52m21.998175018s) [upload=12278]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:32:29.984781139 (11h52m51.728838008s) [upload=12278]: Next segment err: context deadline exceeded Aug 7 21:32:29.984827795 (11h52m51.728884645s) [upload=12278]: Done uploading segments... Aug 7 21:32:29.984851910 (11h52m51.728908763s) [upload=12278]: Upload finished. err: context deadline exceeded Aug 7 21:32:35.321237619 (11h52m57.065294469s) [upload=12287]: Starting upload Aug 7 21:32:35.321310476 (11h52m57.065367326s) [upload=12287]: Done starting upload Aug 7 21:32:35.321418401 (11h52m57.06547525s) [upload=12287]: Uploading segments... Aug 7 21:32:35.321610690 (11h52m57.06566754s) [upload=12287]: Got next segment. Inline: false Aug 7 21:32:35.321654358 (11h52m57.065711209s) [upload=12287]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:35.321659987 (11h52m57.065716847s) [upload=12287]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:05.046284716 (11h53m26.790341587s) [upload=12287]: Next segment err: context deadline exceeded Aug 7 21:33:05.046403894 (11h53m26.790460755s) [upload=12287]: Done uploading segments... Aug 7 21:33:05.046426465 (11h53m26.790483318s) [upload=12287]: Upload finished. err: context deadline exceeded Aug 7 21:33:21.055953150 (11h53m42.80001s) [upload=12296]: Starting upload Aug 7 21:33:21.056022172 (11h53m42.800079036s) [upload=12296]: Done starting upload Aug 7 21:33:21.056266284 (11h53m42.800323139s) [upload=12296]: Uploading segments... Aug 7 21:33:21.056360175 (11h53m42.800417035s) [upload=12296]: Got next segment. Inline: false Aug 7 21:33:21.056419681 (11h53m42.800476544s) [upload=12296]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:21.056426657 (11h53m42.800483517s) [upload=12296]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:50.766946667 (11h54m12.511003536s) [upload=12296]: Next segment err: context deadline exceeded Aug 7 21:33:50.767114018 (11h54m12.511170869s) [upload=12296]: Done uploading segments... Aug 7 21:33:50.767133351 (11h54m12.511190205s) [upload=12296]: Upload finished. err: context deadline exceeded Aug 7 21:34:24.125593602 (11h54m45.869650451s) [upload=12312]: Starting upload Aug 7 21:34:24.125684240 (11h54m45.869741107s) [upload=12312]: Done starting upload Aug 7 21:34:24.125788426 (11h54m45.86984528s) [upload=12312]: Uploading segments... Aug 7 21:34:24.126065319 (11h54m45.870122183s) [upload=12312]: Got next segment. Inline: false Aug 7 21:34:24.126124945 (11h54m45.870181799s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:24.126132194 (11h54m45.870189062s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:34:53.850914976 (11h55m15.594971842s) [upload=12312]: Next segment err: context deadline exceeded Aug 7 21:34:53.851009806 (11h55m15.595066658s) [upload=12312]: Done uploading segments... Aug 7 21:34:53.851029858 (11h55m15.595086713s) [upload=12312]: Upload finished. err: context deadline exceeded Aug 7 21:35:29.693652702 (11h55m51.437709569s) [upload=12327]: Starting upload Aug 7 21:35:29.693720941 (11h55m51.437777791s) [upload=12327]: Done starting upload Aug 7 21:35:29.693804503 (11h55m51.437861352s) [upload=12327]: Uploading segments... Aug 7 21:35:29.694064305 (11h55m51.438121165s) [upload=12327]: Got next segment. Inline: false Aug 7 21:35:29.694379029 (11h55m51.438435891s) [upload=12327]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:29.694386167 (11h55m51.438458876s) [upload=12327]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:59.409055375 (11h56m21.153112257s) [upload=12327]: Next segment err: context deadline exceeded Aug 7 21:35:59.409189915 (11h56m21.153246765s) [upload=12327]: Done uploading segments... Aug 7 21:35:59.409209384 (11h56m21.153266238s) [upload=12327]: Upload finished. err: context deadline exceeded Aug 7 21:36:02.108550882 (11h56m23.852607738s) [upload=12339]: Starting upload Aug 7 21:36:02.108641262 (11h56m23.852698124s) [upload=12339]: Done starting upload Aug 7 21:36:02.108822193 (11h56m23.852879056s) [upload=12339]: Uploading segments... Aug 7 21:36:02.109105776 (11h56m23.853162646s) [upload=12339]: Got next segment. Inline: false Aug 7 21:36:02.109242745 (11h56m23.853299596s) [upload=12339]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:02.109251336 (11h56m23.853308191s) [upload=12339]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:36:31.832122108 (11h56m53.576178974s) [upload=12339]: Next segment err: context deadline exceeded Aug 7 21:36:31.832144427 (11h56m53.576201288s) [upload=12339]: Done uploading segments... Aug 7 21:36:31.832164764 (11h56m53.576221614s) [upload=12339]: Upload finished. err: context deadline exceeded Aug 7 21:36:36.145108941 (11h56m57.889165792s) [upload=12346]: Starting upload Aug 7 21:36:36.145194386 (11h56m57.889251256s) [upload=12346]: Done starting upload Aug 7 21:36:36.145300908 (11h56m57.889357769s) [upload=12346]: Uploading segments... Aug 7 21:36:36.145513778 (11h56m57.88957065s) [upload=12346]: Got next segment. Inline: false Aug 7 21:36:36.145570517 (11h56m57.889627388s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:36.145595053 (11h56m57.88965192s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:37:05.869557094 (11h57m27.613613964s) [upload=12346]: Next segment err: context deadline exceeded Aug 7 21:37:05.869719934 (11h57m27.613776804s) [upload=12346]: Done uploading segments... Aug 7 21:37:05.869768890 (11h57m27.613825737s) [upload=12346]: Upload finished. err: context deadline exceeded Aug 7 21:37:34.228977016 (11h57m55.973033863s) [upload=12361]: Starting upload Aug 7 21:37:34.229041553 (11h57m55.973098425s) [upload=12361]: Done starting upload Aug 7 21:37:34.229357552 (11h57m55.973414413s) [upload=12361]: Uploading segments... Aug 7 21:37:34.229441526 (11h57m55.973498392s) [upload=12361]: Got next segment. Inline: false Aug 7 21:37:34.229523182 (11h57m55.973580053s) [upload=12361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:34.229532143 (11h57m55.97358901s) [upload=12361]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:38:03.955400832 (11h58m25.699457693s) [upload=12361]: Next segment err: context deadline exceeded Aug 7 21:38:03.955719318 (11h58m25.699776168s) [upload=12361]: Done uploading segments... Aug 7 21:38:03.955748514 (11h58m25.699805368s) [upload=12361]: Upload finished. err: context deadline exceeded Aug 7 21:38:17.750476117 (11h58m39.494532967s) [upload=12367]: Starting upload Aug 7 21:38:17.750579920 (11h58m39.494636767s) [upload=12367]: Done starting upload Aug 7 21:38:17.750709067 (11h58m39.494765913s) [upload=12367]: Uploading segments... Aug 7 21:38:17.751240368 (11h58m39.495297269s) [upload=12367]: Got next segment. Inline: false Aug 7 21:38:17.751337897 (11h58m39.495394752s) [upload=12367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:17.751345904 (11h58m39.49540276s) [upload=12367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:38:47.482628669 (11h59m9.226685536s) [upload=12367]: Next segment err: context deadline exceeded Aug 7 21:38:47.482702344 (11h59m9.226759204s) [upload=12367]: Done uploading segments... Aug 7 21:38:47.482721699 (11h59m9.226778549s) [upload=12367]: Upload finished. err: context deadline exceeded Aug 7 21:38:47.921712976 (11h59m9.665769843s) [upload=12376]: Starting upload Aug 7 21:38:47.921816019 (11h59m9.665872884s) [upload=12376]: Done starting upload Aug 7 21:38:47.921973230 (11h59m9.666030093s) [upload=12376]: Uploading segments... Aug 7 21:38:47.922447667 (11h59m9.666504541s) [upload=12376]: Got next segment. Inline: false Aug 7 21:38:47.922523948 (11h59m9.666580816s) [upload=12376]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:47.922530237 (11h59m9.666587106s) [upload=12376]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:39:17.651401827 (11h59m39.395458685s) [upload=12376]: Next segment err: context deadline exceeded Aug 7 21:39:17.651653935 (11h59m39.395710786s) [upload=12376]: Done uploading segments... Aug 7 21:39:17.651678249 (11h59m39.395735101s) [upload=12376]: Upload finished. err: context deadline exceeded
Duration: 30.001653938s
core
uplink
us-select-1
|