Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
1.016472835s
FAIL (2025-08-07 10:52:38 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 1.016501685s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
860.432454ms
FAIL (2025-08-07 10:51:34 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 860.475158ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.327325592s
FAIL (2025-08-07 10:50:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.327375346s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
937.160326ms
FAIL (2025-08-07 10:50:32 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 937.180263ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
2.381937387s
FAIL (2025-08-07 10:50:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.381967767s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
2.530292848s
FAIL (2025-08-07 10:49:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.53031977s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
979.039047ms
FAIL (2025-08-07 10:49:19 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 979.069317ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
3.281739136s
FAIL (2025-08-07 10:48:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 3.281774519s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
889.105092ms
FAIL (2025-08-07 10:48:10 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 889.136727ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
1.238001115s
FAIL (2025-08-07 10:47:26 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 1.238037482s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [X...X.......................................................] duration:
2.39608073s
FAIL (2025-08-07 10:47:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.396113684s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X...X......................................................] duration:
2.401124243s
FAIL (2025-08-07 10:46:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.401177653s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
910.121206ms
FAIL (2025-08-07 10:46:15 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 910.141692ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration:
2.801907213s
FAIL (2025-08-07 10:45:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.801941031s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
1.003225989s
FAIL (2025-08-07 10:45:16 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 1.00325887s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X...X....................................................] duration:
2.791743123s
FAIL (2025-08-07 10:45:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.791776847s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
981.742578ms
FAIL (2025-08-07 10:44:31 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 981.792269ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
896.576421ms
FAIL (2025-08-07 10:44:09 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 896.623562ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X...X...................................................] duration:
2.708493589s
FAIL (2025-08-07 10:44:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.708550799s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
932.617041ms
FAIL (2025-08-07 10:43:17 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 932.645037ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X...X..................................................] duration:
2.321781064s
FAIL (2025-08-07 10:43:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.321821529s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
1.716398829s
FAIL (2025-08-07 10:42:33 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 1.716457065s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X...X.................................................] duration:
2.401288514s
FAIL (2025-08-07 10:42:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.401326801s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X...X................................................] duration:
2.562439295s
FAIL (2025-08-07 10:41:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.56248632s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
1.34124971s
FAIL (2025-08-07 10:41:20 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 1.341278082s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X...X...............................................] duration:
2.676713107s
FAIL (2025-08-07 10:41:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.676755664s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
1.198316714s
FAIL (2025-08-07 10:40:51 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 1.198344477s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X...X..............................................] duration:
2.984812278s
FAIL (2025-08-07 10:40:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.984845861s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
1.510778859s
FAIL (2025-08-07 10:40:00 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 1.510859102s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
895.696401ms
FAIL (2025-08-07 10:39:25 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 895.744065ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration:
2.818125738s
FAIL (2025-08-07 10:39:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.818146902s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
1.21880135s
FAIL (2025-08-07 10:38:27 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 1.218828893s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X...X............................................] duration:
2.595504994s
FAIL (2025-08-07 10:38:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.595540753s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
861.707338ms
FAIL (2025-08-07 10:37:31 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 861.746732ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X...X...........................................] duration:
2.452111046s
FAIL (2025-08-07 10:37:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.452149273s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
970.000834ms
FAIL (2025-08-07 10:36:42 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 970.102011ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
1.103646446s
FAIL (2025-08-07 10:35:53 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 1.10368142s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration:
2.471995189s
FAIL (2025-08-07 10:35:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.472024286s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration:
2.437852547s
FAIL (2025-08-07 10:35:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.437883792s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
852.142242ms
FAIL (2025-08-07 10:34:55 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 852.170424ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............X...X........................................] duration:
2.792729867s
FAIL (2025-08-07 10:34:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.792755926s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
1.471001296s
FAIL (2025-08-07 10:34:38 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 1.471044864s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X...X.......................................] duration:
2.456084565s
FAIL (2025-08-07 10:34:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.456165421s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
1.105815164s
FAIL (2025-08-07 10:33:42 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 1.105845105s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X...X......................................] duration:
2.574972415s
FAIL (2025-08-07 10:33:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.575008012s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
923.177502ms
FAIL (2025-08-07 10:33:01 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 923.20564ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration:
2.479347548s
FAIL (2025-08-07 10:32:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.479392375s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X...X....................................] duration:
2.406675916s
FAIL (2025-08-07 10:31:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.406715634s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
847.086468ms
FAIL (2025-08-07 10:31:49 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== ext segment. Inline: true Aug 7 10:22:08.930607753 (42m30.674664619s) [upload=736]: Waiting for error group managing segments... Aug 7 10:22:08.930654041 (42m30.67471089s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:08.930659735 (42m30.674716603s) [upload=736]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:08.930663415 (42m30.67472029s) [upload=736]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:09.091497540 (42m30.8355544s) [upload=736]: Done uploading segments... Aug 7 10:22:09.091515598 (42m30.835572449s) [upload=736]: Upload finished. err: <nil> Aug 7 10:22:40.035320711 (43m1.779377559s) [upload=745]: Starting upload Aug 7 10:22:40.035385319 (43m1.779442189s) [upload=745]: Done starting upload Aug 7 10:22:40.035464093 (43m1.77952094s) [upload=745]: Uploading segments... Aug 7 10:22:40.035516406 (43m1.779573278s) [upload=745]: Got next segment. Inline: true Aug 7 10:22:40.035521681 (43m1.779578542s) [upload=745]: Waiting for error group managing segments... Aug 7 10:22:40.035570034 (43m1.77962688s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:22:40.035576036 (43m1.779632901s) [upload=745]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:22:40.035579941 (43m1.779636804s) [upload=745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:22:40.202851414 (43m1.946908267s) [upload=745]: Done uploading segments... Aug 7 10:22:40.202869676 (43m1.946926527s) [upload=745]: Upload finished. err: <nil> Aug 7 10:23:54.868005690 (44m16.612062551s) [upload=767]: Starting upload Aug 7 10:23:54.868215067 (44m16.612271931s) [upload=767]: Done starting upload Aug 7 10:23:54.868324545 (44m16.612381395s) [upload=767]: Uploading segments... Aug 7 10:23:54.868399095 (44m16.612455964s) [upload=767]: Got next segment. Inline: true Aug 7 10:23:54.868405374 (44m16.612462234s) [upload=767]: Waiting for error group managing segments... Aug 7 10:23:54.868486917 (44m16.61254377s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:23:54.868495801 (44m16.612552669s) [upload=767]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:23:54.868500456 (44m16.612557323s) [upload=767]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:23:55.070821085 (44m16.814877942s) [upload=767]: Done uploading segments... Aug 7 10:23:55.070843878 (44m16.814900724s) [upload=767]: Upload finished. err: <nil> Aug 7 10:24:26.684213003 (44m48.428269854s) [upload=774]: Starting upload Aug 7 10:24:26.684286839 (44m48.428343699s) [upload=774]: Done starting upload Aug 7 10:24:26.684364512 (44m48.428421375s) [upload=774]: Uploading segments... Aug 7 10:24:26.684412902 (44m48.428469771s) [upload=774]: Got next segment. Inline: true Aug 7 10:24:26.684420284 (44m48.428477135s) [upload=774]: Waiting for error group managing segments... Aug 7 10:24:26.684467924 (44m48.428524784s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:26.684473292 (44m48.428530153s) [upload=774]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:26.684477161 (44m48.428534014s) [upload=774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:26.915664694 (44m48.659721558s) [upload=774]: Done uploading segments... Aug 7 10:24:26.915688586 (44m48.659745448s) [upload=774]: Upload finished. err: <nil> Aug 7 10:24:46.796304085 (45m8.540360957s) [upload=783]: Starting upload Aug 7 10:24:46.796390055 (45m8.540446915s) [upload=783]: Done starting upload Aug 7 10:24:46.796458506 (45m8.540515359s) [upload=783]: Uploading segments... Aug 7 10:24:46.796505775 (45m8.540562645s) [upload=783]: Got next segment. Inline: true Aug 7 10:24:46.796511267 (45m8.540568127s) [upload=783]: Waiting for error group managing segments... Aug 7 10:24:46.796557518 (45m8.540614378s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:24:46.796562658 (45m8.540619519s) [upload=783]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:24:46.796565995 (45m8.540622855s) [upload=783]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:24:47.043220917 (45m8.787277783s) [upload=783]: Done uploading segments... Aug 7 10:24:47.043259854 (45m8.787316718s) [upload=783]: Upload finished. err: <nil> Aug 7 10:25:57.691417285 (46m19.435474138s) [upload=803]: Starting upload Aug 7 10:25:57.691489985 (46m19.435546845s) [upload=803]: Done starting upload Aug 7 10:25:57.691613308 (46m19.435670163s) [upload=803]: Uploading segments... Aug 7 10:25:57.691694715 (46m19.435751584s) [upload=803]: Got next segment. Inline: true Aug 7 10:25:57.691700935 (46m19.435757795s) [upload=803]: Waiting for error group managing segments... Aug 7 10:25:57.691774510 (46m19.435831373s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:25:57.691780933 (46m19.435837784s) [upload=803]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:25:57.691784640 (46m19.435841501s) [upload=803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:25:58.013128364 (46m19.757185224s) [upload=803]: Done uploading segments... Aug 7 10:25:58.013146780 (46m19.757203634s) [upload=803]: Upload finished. err: <nil> Aug 7 10:26:46.591364292 (47m8.335421137s) [upload=817]: Starting upload Aug 7 10:26:46.591414365 (47m8.335471228s) [upload=817]: Done starting upload Aug 7 10:26:46.591490335 (47m8.335547182s) [upload=817]: Uploading segments... Aug 7 10:26:46.591532428 (47m8.335589293s) [upload=817]: Got next segment. Inline: true Aug 7 10:26:46.591536992 (47m8.335593855s) [upload=817]: Waiting for error group managing segments... Aug 7 10:26:46.591576930 (47m8.335633777s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:26:46.591581302 (47m8.335638165s) [upload=817]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:26:46.591583864 (47m8.335640727s) [upload=817]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:26:47.022543716 (47m8.766600579s) [upload=817]: Done uploading segments... Aug 7 10:26:47.022569551 (47m8.766626415s) [upload=817]: Upload finished. err: <nil> Aug 7 10:27:20.835167791 (47m42.579224652s) [upload=827]: Starting upload Aug 7 10:27:20.835268367 (47m42.57932523s) [upload=827]: Done starting upload Aug 7 10:27:20.835355878 (47m42.579412729s) [upload=827]: Uploading segments... Aug 7 10:27:20.835427671 (47m42.579484541s) [upload=827]: Got next segment. Inline: true Aug 7 10:27:20.835434392 (47m42.579491256s) [upload=827]: Waiting for error group managing segments... Aug 7 10:27:20.835496416 (47m42.579553266s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:27:20.835501963 (47m42.579558815s) [upload=827]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:27:20.835505601 (47m42.579562462s) [upload=827]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:27:21.120423971 (47m42.864480824s) [upload=827]: Done uploading segments... Aug 7 10:27:21.120445363 (47m42.864502219s) [upload=827]: Upload finished. err: <nil> Aug 7 10:28:27.352689442 (48m49.096746297s) [upload=849]: Starting upload Aug 7 10:28:27.352757406 (48m49.096814267s) [upload=849]: Done starting upload Aug 7 10:28:27.352842961 (48m49.096899811s) [upload=849]: Uploading segments... Aug 7 10:28:27.352930652 (48m49.096987517s) [upload=849]: Got next segment. Inline: true Aug 7 10:28:27.352937204 (48m49.096994064s) [upload=849]: Waiting for error group managing segments... Aug 7 10:28:27.353004367 (48m49.097061218s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:28:27.353011574 (48m49.097068426s) [upload=849]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:28:27.353015280 (48m49.097072131s) [upload=849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:28:27.646335343 (48m49.390392193s) [upload=849]: Done uploading segments... Aug 7 10:28:27.646354101 (48m49.390410961s) [upload=849]: Upload finished. err: <nil> Aug 7 10:29:05.713625010 (49m27.45768187s) [upload=858]: Starting upload Aug 7 10:29:05.713670258 (49m27.457727107s) [upload=858]: Done starting upload Aug 7 10:29:05.713738123 (49m27.457794974s) [upload=858]: Uploading segments... Aug 7 10:29:05.713779348 (49m27.457836217s) [upload=858]: Got next segment. Inline: true Aug 7 10:29:05.713784034 (49m27.457840893s) [upload=858]: Waiting for error group managing segments... Aug 7 10:29:05.713824768 (49m27.457881618s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:05.713831372 (49m27.457888223s) [upload=858]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:05.713834931 (49m27.457891781s) [upload=858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:05.949211308 (49m27.693268159s) [upload=858]: Done uploading segments... Aug 7 10:29:05.949229910 (49m27.693286763s) [upload=858]: Upload finished. err: <nil> Aug 7 10:29:27.265784503 (49m49.009841354s) [upload=865]: Starting upload Aug 7 10:29:27.265834988 (49m49.009891854s) [upload=865]: Done starting upload Aug 7 10:29:27.265893610 (49m49.009950464s) [upload=865]: Uploading segments... Aug 7 10:29:27.265980157 (49m49.01003702s) [upload=865]: Got next segment. Inline: true Aug 7 10:29:27.265985556 (49m49.010042416s) [upload=865]: Waiting for error group managing segments... Aug 7 10:29:27.266049830 (49m49.010106695s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 10:29:27.266056092 (49m49.010112958s) [upload=865]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 10:29:27.266059647 (49m49.010116511s) [upload=865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 10:29:28.450380434 (49m50.194437294s) [upload=865]: Done uploading segments... Aug 7 10:29:28.450406421 (49m50.194463268s) [upload=865]: Upload finished. err: <nil>
Duration: 847.117755ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................X...X...................................] duration:
2.421203728s
FAIL (2025-08-07 10:31:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
VJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304068432 (22m30.0481253s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.304764616 (22m30.048821486s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305014345 (22m30.049071208s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305477179 (22m30.049534041s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.305880205 (22m30.049937065s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306158266 (22m30.050215143s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306643910 (22m30.050700776s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.306783157 (22m30.050840018s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307230497 (22m30.051287358s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307491498 (22m30.051548362s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.307981781 (22m30.052038642s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308389560 (22m30.052446433s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.308652459 (22m30.052709331s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309135547 (22m30.053192407s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.309257766 (22m30.053314647s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Uploading piece...
Aug 7 10:02:08.874893186 (22m30.618950054s) [upload=380]: Next returned nil segment
Aug 7 10:02:08.874915097 (22m30.618971972s) [upload=380]: Waiting for error group managing segments...
Aug 7 10:02:09.005225928 (22m30.749282796s) [noise=true , address=45.91.116.7:4017 , piece_id=E5GLWCGDFY626F5E3GDLS2TCA6GK5MYOS2V5KIQNHC5ZE5YLDIUQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.010760006 (22m30.754816867s) [noise=true , address=45.91.116.4:4945 , piece_id=PF6BROBDUO7JPAXASEQDFFQFORTKYK3AVREF6FA3X7UUOXM2GBIQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.021031093 (22m30.765087954s) [noise=true , address=45.91.116.14:4214 , piece_id=LMGKPMJEQD7E3VECR3EVWL3AVY23AGWN4CZGVZFAFCDU6AZQA66Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.027507797 (22m30.771564666s) [noise=true , address=45.91.116.8:4051 , piece_id=KTAYSDZ44536HQ67WQX5L6P4YOODNC4T7JYJM6W5CDTBF4JK7RLA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.028579515 (22m30.772636362s) [noise=true , address=45.91.116.6:4996 , piece_id=5IUK44D63DMU224SUDSCBRUE5LVXQ3CLCY66CC4NJ2PN4GKFAJHA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.036714968 (22m30.780771835s) [noise=true , address=104.36.82.62:4440 , piece_id=AKNRS7VRYKCVLQ26RDVS32OIKBVQH757VMZSIH6BUKCM6XPSGMJQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.038707196 (22m30.782764064s) [noise=true , address=104.36.82.47:4534 , piece_id=U6YYACSYCDRWKW5ZZSD3LUS6CKC3D5VPOJNYGW2MVKK4HAGM4D6Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043195674 (22m30.787252526s) [noise=true , address=104.36.82.44:4902 , piece_id=XUXM6X5EUMYQ5GIPDJBHWVFAVI5MOLVYBQFLNNQQB4AIBCTZQ4FQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.043314117 (22m30.787370987s) [noise=true , address=104.36.82.16:4304 , piece_id=YV4R4DJZASOOKKNXQL77GGAATKC7ZN6PVJWPCVJ6J5B35T37W54Q, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.053762161 (22m30.79781901s) [noise=true , address=104.36.82.45:4913 , piece_id=QIVRGZ42GACITRLL7D7KN3BMQ77SE7IST4T2KL25KWHNPTQHKYOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.061702245 (22m30.805759115s) [noise=true , address=104.36.82.61:4377 , piece_id=MXXXUKIOPFXTUUVECENSPA6646VYR232H6QHNL4QIH3EVOFYOH6A, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.064101087 (22m30.808157953s) [noise=true , address=104.36.82.5:4859 , piece_id=S56ZY6BRF67XOCQ5NGUYVY3PNAHGUO2S7VC3S7ALV5XEKSY3AIFA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.066217211 (22m30.810274061s) [noise=true , address=104.36.82.12:4184 , piece_id=7UUGLPE265H3PZRYMYTXXPDCP4NOZA2AQCW3JYPV4TXLP3JT3ZMA, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073243312 (22m30.817300179s) [noise=true , address=45.91.116.11:4135 , piece_id=G7Z2E6JAEEYCA66WPXOU3JA7GKYGU4Z7TWPPBTTMWEX67UCGAKAQ, seg_pos={0 0}, upload=380]: Done uploading piece. err: <nil>
Aug 7 10:02:09.073272803 (22m30.817329668s) [seg_pos={0 0}, upload=380]: Segment reached optimal threshold of 14 pieces.
Aug 7 10:02:09.073773138 (22m30.817830004s) [noise=true , address=104.36.82.46:4481 , piece_id=DGFE47MKNU6IOC6U27PUI7K5VIMDTDL254RDHGXQ5KWI7A72ESYA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Aug 7 10:02:09.073996336 (22m30.818053198s) [noise=true , address=64.49.13.140:4505 , piece_id=KXLPZD7BC6EIINZG52FB4QRJZL2H6AKHVM3CQSKI2FXSD367QLOA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AbFiw33iRaTCNbSt9NmCYG1KBqys85G5VcgXndydkzgnxN4L4): protocol: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45692->64.49.13.140:4505: use of closed network connection
Aug 7 10:02:09.074043859 (22m30.818100721s) [noise=true , address=64.49.13.154:4928 , piece_id=BMHYHXDJUB4A7MCSEYT2MJP67JAZBA75WCRR5IHWPP2R4LNSHXPA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126yMXrtihbzi2iQB399nmL6ARMD7NJp6Bg1YYhhVPJ7kE1DQSD): protocol: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32956->64.49.13.154:4928: use of closed network connection
Aug 7 10:02:09.074110024 (22m30.818166887s) [noise=true , address=64.49.13.138:4487 , piece_id=RQI26IPK4BAD7CBF6QT3UMNU7FI7675BSJPBSKQBE5JGN5OM3CSA, seg_pos={0 0}, upload=380]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42096->64.49.13.138:4487: use of closed network connection
Aug 7 10:02:09.074136418 (22m30.818193282s) [seg_pos={0 0}, upload=380]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 10:02:09.074239505 (22m30.818296369s) [upload=380]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 10:02:09.074246041 (22m30.818302902s) [upload=380]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 10:02:09.221147229 (22m30.965204101s) [upload=380]: Done uploading segments...
Aug 7 10:02:09.221188224 (22m30.965245092s) [upload=380]: Upload finished. err: <nil>
Aug 7 10:03:03.703687023 (23m25.447743892s) [upload=398]: Starting upload
Aug 7 10:03:03.703770546 (23m25.447827427s) [upload=398]: Done starting upload
Aug 7 10:03:03.703902060 (23m25.447958915s) [upload=398]: Uploading segments...
Aug 7 10:03:03.704413533 (23m25.448470395s) [upload=398]: Got next segment. Inline: false
Aug 7 10:03:03.704483839 (23m25.448540718s) [upload=398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 10:03:03.704510589 (23m25.448567459s) [upload=398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 10:03:03.792999911 (23m25.537056781s) [upload=398]: Next segment err: metaclient: Too Many Requests
Aug 7 10:03:03.793060680 (23m25.537117544s) [upload=398]: Done uploading segments...
Aug 7 10:03:03.793084895 (23m25.537141753s) [upload=398]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.421230825s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|