Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 10 out of 60 is failed (threshold=3) [X..XX.X...................................X...XX.X...XX.....] duration: 984.135391ms FAIL (2025-08-08 00:03:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 140355 (14h13m13.036197215s) [upload=14641]: Starting upload Aug 7 23:52:51.292216133 (14h13m13.036272985s) [upload=14641]: Done starting upload Aug 7 23:52:51.292307446 (14h13m13.036364307s) [upload=14641]: Uploading segments... Aug 7 23:52:51.292389087 (14h13m13.036445956s) [upload=14641]: Got next segment. Inline: true Aug 7 23:52:51.292394933 (14h13m13.036451793s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:52:51.292456032 (14h13m13.036512883s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:51.292461787 (14h13m13.036518647s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:51.292467676 (14h13m13.036524537s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:51.436915828 (14h13m13.180972678s) [upload=14641]: Done uploading segments... Aug 7 23:52:51.436934458 (14h13m13.18099131s) [upload=14641]: Upload finished. err: <nil> Aug 7 23:53:10.946261153 (14h13m32.690318028s) [upload=14646]: Starting upload Aug 7 23:53:10.946350198 (14h13m32.690407065s) [upload=14646]: Done starting upload Aug 7 23:53:10.946435423 (14h13m32.690492276s) [upload=14646]: Uploading segments... Aug 7 23:53:10.946489255 (14h13m32.690546124s) [upload=14646]: Got next segment. Inline: true Aug 7 23:53:10.946497660 (14h13m32.690554519s) [upload=14646]: Waiting for error group managing segments... Aug 7 23:53:10.946542678 (14h13m32.690599529s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:10.946548267 (14h13m32.690605128s) [upload=14646]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:10.946553792 (14h13m32.690610652s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:14.554651436 (14h13m36.298708309s) [upload=14646]: Done uploading segments... Aug 7 23:53:14.554675183 (14h13m36.298732045s) [upload=14646]: Upload finished. err: <nil> Aug 7 23:53:53.584502029 (14h14m15.328558897s) [upload=14658]: Starting upload Aug 7 23:53:53.584599757 (14h14m15.328656616s) [upload=14658]: Done starting upload Aug 7 23:53:53.584702364 (14h14m15.328759218s) [upload=14658]: Uploading segments... Aug 7 23:53:53.584798610 (14h14m15.32885548s) [upload=14658]: Got next segment. Inline: true Aug 7 23:53:53.584806594 (14h14m15.328863461s) [upload=14658]: Waiting for error group managing segments... Aug 7 23:53:53.584875193 (14h14m15.328932062s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:53.584882324 (14h14m15.328939193s) [upload=14658]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:53.584901657 (14h14m15.328958528s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:23.318821682 (14h14m45.062878542s) [upload=14658]: Done uploading segments... Aug 7 23:54:23.318863033 (14h14m45.062919884s) [upload=14658]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:36.775209359 (14h14m58.519266209s) [upload=14671]: Starting upload Aug 7 23:54:36.775292235 (14h14m58.519349112s) [upload=14671]: Done starting upload Aug 7 23:54:36.775412031 (14h14m58.519468883s) [upload=14671]: Uploading segments... Aug 7 23:54:36.775480767 (14h14m58.519537636s) [upload=14671]: Got next segment. Inline: true Aug 7 23:54:36.775487320 (14h14m58.519544181s) [upload=14671]: Waiting for error group managing segments... Aug 7 23:54:36.775539752 (14h14m58.519596612s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:36.775545893 (14h14m58.519602753s) [upload=14671]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:36.775549619 (14h14m58.519606479s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:36.926276427 (14h14m58.670333277s) [upload=14671]: Done uploading segments... Aug 7 23:54:36.926295206 (14h14m58.670352057s) [upload=14671]: Upload finished. err: <nil> Aug 7 23:54:57.675347843 (14h15m19.419404694s) [upload=14676]: Starting upload Aug 7 23:54:57.675412352 (14h15m19.419469212s) [upload=14676]: Done starting upload Aug 7 23:54:57.675475670 (14h15m19.41953252s) [upload=14676]: Uploading segments... Aug 7 23:54:57.675516336 (14h15m19.419573205s) [upload=14676]: Got next segment. Inline: true Aug 7 23:54:57.675521071 (14h15m19.419577931s) [upload=14676]: Waiting for error group managing segments... Aug 7 23:54:57.675565512 (14h15m19.419622375s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:57.675574255 (14h15m19.419631202s) [upload=14676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:57.675580694 (14h15m19.419637563s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:27.407552485 (14h15m49.151609344s) [upload=14676]: Done uploading segments... Aug 7 23:55:27.407592978 (14h15m49.151649829s) [upload=14676]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:06.851646232 (14h16m28.595703082s) [upload=14696]: Starting upload Aug 7 23:56:06.851717330 (14h16m28.595774191s) [upload=14696]: Done starting upload Aug 7 23:56:06.851786008 (14h16m28.59584286s) [upload=14696]: Uploading segments... Aug 7 23:56:06.851834232 (14h16m28.595891101s) [upload=14696]: Got next segment. Inline: true Aug 7 23:56:06.851839675 (14h16m28.595896535s) [upload=14696]: Waiting for error group managing segments... Aug 7 23:56:06.851896275 (14h16m28.595953125s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:06.851901659 (14h16m28.59595852s) [upload=14696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:06.851905505 (14h16m28.595962364s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:07.006202008 (14h16m28.750258868s) [upload=14696]: Done uploading segments... Aug 7 23:56:07.006222000 (14h16m28.750278861s) [upload=14696]: Upload finished. err: <nil> Aug 7 23:56:59.020763574 (14h17m20.764820421s) [upload=14712]: Starting upload Aug 7 23:56:59.020858955 (14h17m20.764915806s) [upload=14712]: Done starting upload Aug 7 23:56:59.020935444 (14h17m20.764992295s) [upload=14712]: Uploading segments... Aug 7 23:56:59.021007501 (14h17m20.76506437s) [upload=14712]: Got next segment. Inline: true Aug 7 23:56:59.021013631 (14h17m20.765070491s) [upload=14712]: Waiting for error group managing segments... Aug 7 23:56:59.021070471 (14h17m20.76512733s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:59.021076966 (14h17m20.765133821s) [upload=14712]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:59.021081305 (14h17m20.76513816s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:59.154820708 (14h17m20.898877569s) [upload=14712]: Done uploading segments... Aug 7 23:56:59.154839835 (14h17m20.898896685s) [upload=14712]: Upload finished. err: <nil> Aug 7 23:58:10.963677657 (14h18m32.707734508s) [upload=14728]: Starting upload Aug 7 23:58:10.963770332 (14h18m32.70782718s) [upload=14728]: Done starting upload Aug 7 23:58:10.963851725 (14h18m32.707908586s) [upload=14728]: Uploading segments... Aug 7 23:58:10.963901559 (14h18m32.707958431s) [upload=14728]: Got next segment. Inline: true Aug 7 23:58:10.963907663 (14h18m32.707964525s) [upload=14728]: Waiting for error group managing segments... Aug 7 23:58:10.963960167 (14h18m32.708017032s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:10.963966034 (14h18m32.708022896s) [upload=14728]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:10.963969944 (14h18m32.708026806s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.106069715 (14h18m32.850126571s) [upload=14728]: Done uploading segments... Aug 7 23:58:11.106112099 (14h18m32.850168961s) [upload=14728]: Upload finished. err: <nil> Aug 7 23:59:19.078128070 (14h19m40.822184931s) [upload=14749]: Starting upload Aug 7 23:59:19.078217499 (14h19m40.822274359s) [upload=14749]: Done starting upload Aug 7 23:59:19.078291040 (14h19m40.822347895s) [upload=14749]: Uploading segments... Aug 7 23:59:19.078333863 (14h19m40.822390732s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:19.078339134 (14h19m40.822395984s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:19.078391868 (14h19m40.822448717s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:19.078397213 (14h19m40.822454064s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:19.078401054 (14h19m40.822457905s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:48.808547234 (14h20m10.552604081s) [upload=14749]: Done uploading segments... Aug 7 23:59:48.808568563 (14h20m10.552625409s) [upload=14749]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:20.896255037 (14h20m42.640311905s) [upload=14765]: Starting upload Aug 8 00:00:20.896381648 (14h20m42.640438517s) [upload=14765]: Done starting upload Aug 8 00:00:20.896532416 (14h20m42.640589282s) [upload=14765]: Uploading segments... Aug 8 00:00:20.896610121 (14h20m42.640666994s) [upload=14765]: Got next segment. Inline: true Aug 8 00:00:20.896618785 (14h20m42.640675649s) [upload=14765]: Waiting for error group managing segments... Aug 8 00:00:20.896682486 (14h20m42.640739601s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:20.896690306 (14h20m42.640747187s) [upload=14765]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:20.896694946 (14h20m42.640751814s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:50.607106781 (14h21m12.351163663s) [upload=14765]: Done uploading segments... Aug 8 00:00:50.607134684 (14h21m12.351191558s) [upload=14765]: Upload finished. err: metaclient: context deadline exceeded Duration: 984.16823ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.................................X....] duration: 3.553090785s FAIL (2025-08-08 00:03:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after e4bc51e4c3741c8242928f6ea44cd583e1de2244d7996b5999976d42b698bf65 != before 74576f6679446814770053fb5ecf3ed143c5a0d1682c10852d0fdad56f6f62fd ======LOGS====== em_ObjectBegin Aug 8 00:00:04.218463528 (14h20m25.962520394s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:04.422837522 (14h20m26.166894398s) [seg_pos={0 0}, upload=14759]: Begin upload segment... Aug 8 00:00:04.422913763 (14h20m26.166970631s) [seg_pos={0 0}, upload=14759]: Done begin upload segment. Aug 8 00:00:04.422953631 (14h20m26.167010505s) [noise=true , address=64.49.13.138:4490 , piece_id=VWBCADPZQBKS6HWUJZ6YWK7FDXN6Z2TSBJKK4Z7P3SDYS476UG5A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.423659818 (14h20m26.167716937s) [noise=true , address=104.36.82.38:4713 , piece_id=44IRUAVNYKZVR74WG64GQPGH3J337QYLGYWC3BUOZ3VS5NPJL27A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.424122548 (14h20m26.168179422s) [noise=true , address=79.127.201.214:4295 , piece_id=CCDJBXRBUJ3NL4PDRRCTXBYXWYBIPYSNU3T3I6R4CTJWJRZTGGQA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.424515211 (14h20m26.168572087s) [noise=true , address=104.36.82.19:4404 , piece_id=LHQKAL4W24GGRXIZJ672W6CXY3NDMUN6NJVH2SFRHETKJMEHW62Q, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.424927726 (14h20m26.1689846s) [noise=true , address=104.36.82.3:4796 , piece_id=APJB3DPBNQBOFK34LZ5MPOOR45ZXRQSMFWB4SOCHFOJCSDW3GWVQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.425328696 (14h20m26.169385571s) [noise=true , address=64.49.13.139:4470 , piece_id=WOXVGGQF2Q5PM6V3LGUOMMA4ZU7UPVW2KOR25TQ32PXN6OWCGGLA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.425736715 (14h20m26.169793589s) [noise=true , address=104.36.82.17:4335 , piece_id=TB54EDHJ5XMYQPXWQB2NMUECRW5VTECH7H6KOPLCFUJ5SXS7U4OQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.426503554 (14h20m26.170560432s) [noise=true , address=104.36.82.59:4318 , piece_id=EHUOWZNGTRHSK7HXYEYYREX66ES4MPYKEU34EW66PFUPHI7VS3XQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.430790810 (14h20m26.174847683s) [noise=true , address=64.49.13.155:4916 , piece_id=54U5CI5CPB7DR56LS2GCOWNANROABC65PXTMSSVUTMZDQEZJY7CA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.431475312 (14h20m26.175532189s) [noise=true , address=104.36.82.62:4425 , piece_id=BOX4GUARMRXU5IIE64CELJHQZFT6UWG4TFDATZHX5YFCJTFV2N4A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.431922656 (14h20m26.175979526s) [noise=true , address=104.36.82.58:4268 , piece_id=YFPLOYXHUU3SOZJHFEMSTCNBM55AYJ5NONP7GLTDFU3OTZTW4BIQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.432313695 (14h20m26.176370573s) [noise=true , address=104.36.82.55:4221 , piece_id=G3QN6J3Q4ZSDPAESESJMAWK3ICJLFBWPWY76QGCSMTHKYP4SUJ6Q, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.432740279 (14h20m26.17679715s) [noise=true , address=45.91.116.14:4195 , piece_id=OFPVRGPPQRMNV3FV46GHPFWSXSNSMDDM7MZ7MFO5D4SP7NS6UZGA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.433162962 (14h20m26.177219835s) [noise=true , address=104.36.82.44:4885 , piece_id=ZHPCTPQUPDXJDPZQPBFSOD6KYKAVCHXQNXESSKEXNDU3XSL6OY3A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.433650237 (14h20m26.177707112s) [noise=true , address=104.36.82.43:4854 , piece_id=CY3WAFZSTIQ2AV6DYPIOZSOORUBPL3FBZK4CZNWL3SMOVG7DSZGQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.434316480 (14h20m26.178373354s) [noise=true , address=64.49.13.26:4948 , piece_id=TN7TAN4AYCIDUXZHESWSGMPXYDEM3ZN5RIS5HV3OVOALIFLJDQ7A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.443916849 (14h20m26.187973724s) [noise=true , address=104.36.82.61:4393 , piece_id=MIKYUC5I7GASL3XCYWW7QHLYNJIKSLIHAPNF6NMV45ACHHJLTJJQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.444505957 (14h20m26.188562832s) [noise=true , address=45.91.116.6:4982 , piece_id=M3LYDOCIFKY5YHDBN6ZX6XGSUASZP74QRPZX3VFYSWD37QGHFNBA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:05.200540577 (14h20m26.944597434s) [upload=14759]: Next returned nil segment Aug 8 00:00:05.200687222 (14h20m26.944744097s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:05.355754277 (14h20m27.099811131s) [noise=true , address=104.36.82.59:4318 , piece_id=EHUOWZNGTRHSK7HXYEYYREX66ES4MPYKEU34EW66PFUPHI7VS3XQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.358198826 (14h20m27.102255692s) [noise=true , address=104.36.82.58:4268 , piece_id=YFPLOYXHUU3SOZJHFEMSTCNBM55AYJ5NONP7GLTDFU3OTZTW4BIQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.386961372 (14h20m27.13101822s) [noise=true , address=104.36.82.19:4404 , piece_id=LHQKAL4W24GGRXIZJ672W6CXY3NDMUN6NJVH2SFRHETKJMEHW62Q, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.391547939 (14h20m27.135604807s) [noise=true , address=104.36.82.55:4221 , piece_id=G3QN6J3Q4ZSDPAESESJMAWK3ICJLFBWPWY76QGCSMTHKYP4SUJ6Q, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.391724549 (14h20m27.135781419s) [noise=true , address=104.36.82.43:4854 , piece_id=CY3WAFZSTIQ2AV6DYPIOZSOORUBPL3FBZK4CZNWL3SMOVG7DSZGQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.625387364 (14h20m27.369444214s) [noise=true , address=104.36.82.17:4335 , piece_id=TB54EDHJ5XMYQPXWQB2NMUECRW5VTECH7H6KOPLCFUJ5SXS7U4OQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.685379596 (14h20m27.429436466s) [noise=true , address=104.36.82.38:4713 , piece_id=44IRUAVNYKZVR74WG64GQPGH3J337QYLGYWC3BUOZ3VS5NPJL27A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.696642486 (14h20m27.440699351s) [noise=true , address=45.91.116.6:4982 , piece_id=M3LYDOCIFKY5YHDBN6ZX6XGSUASZP74QRPZX3VFYSWD37QGHFNBA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.722664674 (14h20m27.466721544s) [noise=true , address=104.36.82.3:4796 , piece_id=APJB3DPBNQBOFK34LZ5MPOOR45ZXRQSMFWB4SOCHFOJCSDW3GWVQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.733358308 (14h20m27.477415158s) [noise=true , address=104.36.82.62:4425 , piece_id=BOX4GUARMRXU5IIE64CELJHQZFT6UWG4TFDATZHX5YFCJTFV2N4A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.781882083 (14h20m27.525938951s) [noise=true , address=45.91.116.14:4195 , piece_id=OFPVRGPPQRMNV3FV46GHPFWSXSNSMDDM7MZ7MFO5D4SP7NS6UZGA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.908897662 (14h20m27.652954524s) [noise=true , address=79.127.201.214:4295 , piece_id=CCDJBXRBUJ3NL4PDRRCTXBYXWYBIPYSNU3T3I6R4CTJWJRZTGGQA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:06.052322455 (14h20m27.796379308s) [noise=true , address=64.49.13.139:4470 , piece_id=WOXVGGQF2Q5PM6V3LGUOMMA4ZU7UPVW2KOR25TQ32PXN6OWCGGLA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:06.059613546 (14h20m27.803670401s) [noise=true , address=64.49.13.155:4916 , piece_id=54U5CI5CPB7DR56LS2GCOWNANROABC65PXTMSSVUTMZDQEZJY7CA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:06.059650800 (14h20m27.803707652s) [seg_pos={0 0}, upload=14759]: Segment reached optimal threshold of 14 pieces. Aug 8 00:00:06.061105436 (14h20m27.805162307s) [noise=true , address=64.49.13.138:4490 , piece_id=VWBCADPZQBKS6HWUJZ6YWK7FDXN6Z2TSBJKK4Z7P3SDYS476UG5A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 8 00:00:06.061315764 (14h20m27.805372629s) [noise=true , address=64.49.13.26:4948 , piece_id=TN7TAN4AYCIDUXZHESWSGMPXYDEM3ZN5RIS5HV3OVOALIFLJDQ7A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVV4rAAZRd4DcLsakTTyPRNWck7AjBA6StKvsVLH2dCKVkwHB): protocol: expected piece hash; context canceled; EOF Aug 8 00:00:06.061491394 (14h20m27.805548286s) [noise=true , address=104.36.82.44:4885 , piece_id=ZHPCTPQUPDXJDPZQPBFSOD6KYKAVCHXQNXESSKEXNDU3XSL6OY3A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: write tcp 139.59.144.22:35754->104.36.82.44:4885: use of closed network connection; write tcp 139.59.144.22:35754->104.36.82.44:4885: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35754->104.36.82.44:4885: use of closed network connection Aug 8 00:00:06.078661046 (14h20m27.822717918s) [noise=true , address=104.36.82.61:4393 , piece_id=MIKYUC5I7GASL3XCYWW7QHLYNJIKSLIHAPNF6NMV45ACHHJLTJJQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xfCSdYjRRxMaTw47dQmaXzGmMWyV1oCCGx7JdhvGqaaoN14Ff): protocol: write tcp 139.59.144.22:48102->104.36.82.61:4393: use of closed network connection; write tcp 139.59.144.22:48102->104.36.82.61:4393: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48102->104.36.82.61:4393: use of closed network connection Aug 8 00:00:06.078739080 (14h20m27.822795929s) [seg_pos={0 0}, upload=14759]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:00:06.078890768 (14h20m27.82294764s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:00:06.078899977 (14h20m27.82295685s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:06.568180745 (14h20m28.312237613s) [upload=14759]: Done uploading segments... Aug 8 00:00:06.568206007 (14h20m28.312262863s) [upload=14759]: Upload finished. err: <nil> Duration:* 3.553122729s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................................X] duration: 30.001669068s FAIL (2025-08-08 00:03:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001710564s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [...............................................XX..XX.X.X...] duration: 868.042262ms FAIL (2025-08-08 00:03:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0]: Starting upload Aug 7 23:53:17.153036870 (14h13m38.89709373s) [upload=14650]: Done starting upload Aug 7 23:53:17.153154678 (14h13m38.897211544s) [upload=14650]: Uploading segments... Aug 7 23:53:17.153233089 (14h13m38.897289957s) [upload=14650]: Got next segment. Inline: true Aug 7 23:53:17.153240256 (14h13m38.897297124s) [upload=14650]: Waiting for error group managing segments... Aug 7 23:53:17.153318892 (14h13m38.897375757s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:17.153325816 (14h13m38.89738268s) [upload=14650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:17.153330453 (14h13m38.897387314s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:16.875757982 (14h14m38.619814845s) [upload=14650]: Done uploading segments... Aug 7 23:54:16.875781849 (14h14m38.619838701s) [upload=14650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:24.158448119 (14h14m45.90250498s) [upload=14667]: Starting upload Aug 7 23:54:24.158525946 (14h14m45.902582806s) [upload=14667]: Done starting upload Aug 7 23:54:24.158598972 (14h14m45.902655824s) [upload=14667]: Uploading segments... Aug 7 23:54:24.158654148 (14h14m45.902711017s) [upload=14667]: Got next segment. Inline: true Aug 7 23:54:24.158659995 (14h14m45.902716856s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:54:24.158714896 (14h14m45.902771747s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:24.158720357 (14h14m45.902777209s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:24.158724105 (14h14m45.902780965s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:23.888400190 (14h15m45.63245704s) [upload=14667]: Done uploading segments... Aug 7 23:55:23.888419087 (14h15m45.632475938s) [upload=14667]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:24.297146309 (14h15m46.041203174s) [upload=14687]: Starting upload Aug 7 23:55:24.297232144 (14h15m46.04128901s) [upload=14687]: Done starting upload Aug 7 23:55:24.297315772 (14h15m46.041372619s) [upload=14687]: Uploading segments... Aug 7 23:55:24.297396238 (14h15m46.041453151s) [upload=14687]: Got next segment. Inline: true Aug 7 23:55:24.297403729 (14h15m46.04146059s) [upload=14687]: Waiting for error group managing segments... Aug 7 23:55:24.297444116 (14h15m46.041500976s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:24.297450971 (14h15m46.041507831s) [upload=14687]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:24.297453883 (14h15m46.041510749s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:24.439126204 (14h15m46.183183055s) [upload=14687]: Done uploading segments... Aug 7 23:55:24.439141121 (14h15m46.183197972s) [upload=14687]: Upload finished. err: <nil> Aug 7 23:56:03.068174127 (14h16m24.812230977s) [upload=14695]: Starting upload Aug 7 23:56:03.068258755 (14h16m24.812315617s) [upload=14695]: Done starting upload Aug 7 23:56:03.068368961 (14h16m24.812425812s) [upload=14695]: Uploading segments... Aug 7 23:56:03.068437851 (14h16m24.812494719s) [upload=14695]: Got next segment. Inline: true Aug 7 23:56:03.068445058 (14h16m24.812501927s) [upload=14695]: Waiting for error group managing segments... Aug 7 23:56:03.068516175 (14h16m24.812573049s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:03.068523275 (14h16m24.812580137s) [upload=14695]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:03.068527513 (14h16m24.812584488s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:03.370378590 (14h16m25.114435452s) [upload=14695]: Done uploading segments... Aug 7 23:56:03.370401742 (14h16m25.114458656s) [upload=14695]: Upload finished. err: <nil> Aug 7 23:56:41.636734874 (14h17m3.380791742s) [upload=14707]: Starting upload Aug 7 23:56:41.636844337 (14h17m3.380901201s) [upload=14707]: Done starting upload Aug 7 23:56:41.636938361 (14h17m3.380995223s) [upload=14707]: Uploading segments... Aug 7 23:56:41.637004566 (14h17m3.381061436s) [upload=14707]: Got next segment. Inline: true Aug 7 23:56:41.637011720 (14h17m3.381068581s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:56:41.637073380 (14h17m3.381130242s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:41.637079684 (14h17m3.381136546s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:41.637083892 (14h17m3.381140753s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:41.369074470 (14h18m3.113131318s) [upload=14707]: Done uploading segments... Aug 7 23:57:41.369096313 (14h18m3.11315316s) [upload=14707]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:41.785857243 (14h18m3.529914104s) [upload=14726]: Starting upload Aug 7 23:57:41.785973295 (14h18m3.530030155s) [upload=14726]: Done starting upload Aug 7 23:57:41.786053418 (14h18m3.530110268s) [upload=14726]: Uploading segments... Aug 7 23:57:41.786115407 (14h18m3.530172273s) [upload=14726]: Got next segment. Inline: true Aug 7 23:57:41.786121268 (14h18m3.530178127s) [upload=14726]: Waiting for error group managing segments... Aug 7 23:57:41.786162041 (14h18m3.530218901s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:41.786167247 (14h18m3.530224108s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:41.786170583 (14h18m3.530227443s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.518215592 (14h19m3.262272452s) [upload=14726]: Done uploading segments... Aug 7 23:58:41.518237426 (14h19m3.262294275s) [upload=14726]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:49.080161447 (14h19m10.824218292s) [upload=14741]: Starting upload Aug 7 23:58:49.080236560 (14h19m10.824293407s) [upload=14741]: Done starting upload Aug 7 23:58:49.080313897 (14h19m10.824370744s) [upload=14741]: Uploading segments... Aug 7 23:58:49.080371825 (14h19m10.824428691s) [upload=14741]: Got next segment. Inline: true Aug 7 23:58:49.080376725 (14h19m10.824433586s) [upload=14741]: Waiting for error group managing segments... Aug 7 23:58:49.080418334 (14h19m10.824475197s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:49.080422560 (14h19m10.824479419s) [upload=14741]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:49.080438323 (14h19m10.824495186s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:49.220444289 (14h19m10.964501139s) [upload=14741]: Done uploading segments... Aug 7 23:58:49.220462660 (14h19m10.96451951s) [upload=14741]: Upload finished. err: <nil> Aug 7 23:59:26.271206280 (14h19m48.015263132s) [upload=14752]: Starting upload Aug 7 23:59:26.271300842 (14h19m48.015357706s) [upload=14752]: Done starting upload Aug 7 23:59:26.271409542 (14h19m48.015466389s) [upload=14752]: Uploading segments... Aug 7 23:59:26.271471969 (14h19m48.015528839s) [upload=14752]: Got next segment. Inline: true Aug 7 23:59:26.271478422 (14h19m48.015535286s) [upload=14752]: Waiting for error group managing segments... Aug 7 23:59:26.271524304 (14h19m48.015581168s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.271531803 (14h19m48.01558867s) [upload=14752]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:26.271535740 (14h19m48.015592604s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.006104454 (14h20m47.750161323s) [upload=14752]: Done uploading segments... Aug 8 00:00:26.006140966 (14h20m47.750197816s) [upload=14752]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:26.408809053 (14h20m48.152865927s) [upload=14766]: Starting upload Aug 8 00:00:26.408886561 (14h20m48.152943434s) [upload=14766]: Done starting upload Aug 8 00:00:26.408975679 (14h20m48.15303253s) [upload=14766]: Uploading segments... Aug 8 00:00:26.409028539 (14h20m48.153085476s) [upload=14766]: Got next segment. Inline: true Aug 8 00:00:26.409034850 (14h20m48.153091712s) [upload=14766]: Waiting for error group managing segments... Aug 8 00:00:26.409085190 (14h20m48.153142042s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:26.409090930 (14h20m48.153147793s) [upload=14766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:26.409094742 (14h20m48.153151605s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.549853950 (14h20m48.2939108s) [upload=14766]: Done uploading segments... Aug 8 00:00:26.549878419 (14h20m48.293935265s) [upload=14766]: Upload finished. err: <nil> Aug 8 00:01:08.274172938 (14h21m30.018229799s) [upload=14780]: Starting upload Aug 8 00:01:08.274248443 (14h21m30.018305302s) [upload=14780]: Done starting upload Aug 8 00:01:08.274326739 (14h21m30.018383585s) [upload=14780]: Uploading segments... Aug 8 00:01:08.274380689 (14h21m30.018437549s) [upload=14780]: Got next segment. Inline: true Aug 8 00:01:08.274385704 (14h21m30.018442565s) [upload=14780]: Waiting for error group managing segments... Aug 8 00:01:08.274437335 (14h21m30.018494182s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:08.274442821 (14h21m30.018499667s) [upload=14780]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:08.274445609 (14h21m30.018502454s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:08.005343335 (14h22m29.749400197s) [upload=14780]: Done uploading segments... Aug 8 00:02:08.005365121 (14h22m29.749421982s) [upload=14780]: Upload finished. err: metaclient: context deadline exceeded Duration: 868.072302ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [.X.........................................X.XXX.X.X.....X..] duration: 880.288534ms FAIL (2025-08-08 00:03:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 880.339144ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X.X............................................XXX.X.X......] duration: 33.588211362s FAIL (2025-08-08 00:03:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.588243961s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 120.164039ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-08 00:03:41 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 120.205319ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX........................................X.XX...XX.X..X...] duration: 2.155408758s FAIL (2025-08-08 00:03:32 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.155434845s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.XX.X..............................................X...X....] duration: 1.098771493s FAIL (2025-08-08 00:03:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.098792787s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXX...........X............................XX.X.X...XX......] duration: 2.598157827s FAIL (2025-08-08 00:03:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 6]: Uploading piece... Aug 7 23:57:10.893816491 (14h17m32.637873354s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.894285524 (14h17m32.638342391s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895348932 (14h17m32.639405787s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895932061 (14h17m32.639988921s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.896545320 (14h17m32.640602184s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897117472 (14h17m32.641174341s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897621325 (14h17m32.641678192s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898101338 (14h17m32.642158204s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898693607 (14h17m32.642750474s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.900512475 (14h17m32.644569335s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.902337804 (14h17m32.646394669s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.904215667 (14h17m32.648272537s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.906034481 (14h17m32.650091365s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.907857426 (14h17m32.651914295s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.908666188 (14h17m32.652723069s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:11.510238622 (14h17m33.254295484s) [upload=14716]: Next returned nil segment Aug 7 23:57:11.510261464 (14h17m33.254318324s) [upload=14716]: Waiting for error group managing segments... Aug 7 23:57:11.657094388 (14h17m33.401151236s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.674408374 (14h17m33.418465234s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.681203586 (14h17m33.425260449s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.682780850 (14h17m33.4268377s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.684042315 (14h17m33.428099176s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.719658547 (14h17m33.463715401s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.730348391 (14h17m33.474405242s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.748264399 (14h17m33.492321249s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.769764028 (14h17m33.513820878s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.774177874 (14h17m33.518234733s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.789864472 (14h17m33.533921326s) [noise=true , address=45.91.116.13:4153 , piece_id=WBFFQT5I6ZR5ZKUJRFULNOJZ455KIJMA5EEATDDOCVFUKRD67NFA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.799519597 (14h17m33.543576447s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800753821 (14h17m33.544810667s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800785983 (14h17m33.544842842s) [seg_pos={0 0}, upload=14716]: Segment reached optimal threshold of 13 pieces. Aug 7 23:57:11.801026052 (14h17m33.545082898s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801098020 (14h17m33.545154867s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801174944 (14h17m33.54523179s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection Aug 7 23:57:11.801184926 (14h17m33.545241773s) [seg_pos={0 0}, upload=14716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:57:11.801257096 (14h17m33.545313956s) [upload=14716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:11.801261009 (14h17m33.545317869s) [upload=14716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:11.942784613 (14h17m33.686841513s) [upload=14716]: Done uploading segments... Aug 7 23:57:11.942815424 (14h17m33.686872279s) [upload=14716]: Upload finished. err: <nil> Aug 7 23:58:20.561350310 (14h18m42.305407184s) [upload=14732]: Starting upload Aug 7 23:58:20.561483402 (14h18m42.30554027s) [upload=14732]: Done starting upload Aug 7 23:58:20.561650297 (14h18m42.305707159s) [upload=14732]: Uploading segments... Aug 7 23:58:20.562196341 (14h18m42.306253212s) [upload=14732]: Got next segment. Inline: false Aug 7 23:58:20.562334166 (14h18m42.306391031s) [upload=14732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:20.562343255 (14h18m42.306400125s) [upload=14732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:50.281195862 (14h19m12.025252712s) [upload=14732]: Next segment err: context deadline exceeded Aug 7 23:58:50.281383705 (14h19m12.025440555s) [upload=14732]: Done uploading segments... Aug 7 23:58:50.281405600 (14h19m12.02546245s) [upload=14732]: Upload finished. err: context deadline exceeded Aug 7 23:58:50.716777496 (14h19m12.460834346s) [upload=14742]: Starting upload Aug 7 23:58:50.716857905 (14h19m12.460914751s) [upload=14742]: Done starting upload Aug 7 23:58:50.716999972 (14h19m12.461056825s) [upload=14742]: Uploading segments... Aug 7 23:58:50.717186124 (14h19m12.461242976s) [upload=14742]: Got next segment. Inline: false Aug 7 23:58:50.717245601 (14h19m12.461302466s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:50.717252185 (14h19m12.461309053s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:20.436529435 (14h19m42.180586305s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:20.436734882 (14h19m42.180791733s) [upload=14742]: Done uploading segments... Aug 7 23:59:20.436756200 (14h19m42.180813054s) [upload=14742]: Upload finished. err: context deadline exceeded Duration: 2.598190429s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [................................................XX..XX.X.X..] duration: 872.238479ms FAIL (2025-08-08 00:03:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0]: Starting upload Aug 7 23:53:17.153036870 (14h13m38.89709373s) [upload=14650]: Done starting upload Aug 7 23:53:17.153154678 (14h13m38.897211544s) [upload=14650]: Uploading segments... Aug 7 23:53:17.153233089 (14h13m38.897289957s) [upload=14650]: Got next segment. Inline: true Aug 7 23:53:17.153240256 (14h13m38.897297124s) [upload=14650]: Waiting for error group managing segments... Aug 7 23:53:17.153318892 (14h13m38.897375757s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:17.153325816 (14h13m38.89738268s) [upload=14650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:17.153330453 (14h13m38.897387314s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:16.875757982 (14h14m38.619814845s) [upload=14650]: Done uploading segments... Aug 7 23:54:16.875781849 (14h14m38.619838701s) [upload=14650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:24.158448119 (14h14m45.90250498s) [upload=14667]: Starting upload Aug 7 23:54:24.158525946 (14h14m45.902582806s) [upload=14667]: Done starting upload Aug 7 23:54:24.158598972 (14h14m45.902655824s) [upload=14667]: Uploading segments... Aug 7 23:54:24.158654148 (14h14m45.902711017s) [upload=14667]: Got next segment. Inline: true Aug 7 23:54:24.158659995 (14h14m45.902716856s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:54:24.158714896 (14h14m45.902771747s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:24.158720357 (14h14m45.902777209s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:24.158724105 (14h14m45.902780965s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:23.888400190 (14h15m45.63245704s) [upload=14667]: Done uploading segments... Aug 7 23:55:23.888419087 (14h15m45.632475938s) [upload=14667]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:24.297146309 (14h15m46.041203174s) [upload=14687]: Starting upload Aug 7 23:55:24.297232144 (14h15m46.04128901s) [upload=14687]: Done starting upload Aug 7 23:55:24.297315772 (14h15m46.041372619s) [upload=14687]: Uploading segments... Aug 7 23:55:24.297396238 (14h15m46.041453151s) [upload=14687]: Got next segment. Inline: true Aug 7 23:55:24.297403729 (14h15m46.04146059s) [upload=14687]: Waiting for error group managing segments... Aug 7 23:55:24.297444116 (14h15m46.041500976s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:24.297450971 (14h15m46.041507831s) [upload=14687]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:24.297453883 (14h15m46.041510749s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:24.439126204 (14h15m46.183183055s) [upload=14687]: Done uploading segments... Aug 7 23:55:24.439141121 (14h15m46.183197972s) [upload=14687]: Upload finished. err: <nil> Aug 7 23:56:03.068174127 (14h16m24.812230977s) [upload=14695]: Starting upload Aug 7 23:56:03.068258755 (14h16m24.812315617s) [upload=14695]: Done starting upload Aug 7 23:56:03.068368961 (14h16m24.812425812s) [upload=14695]: Uploading segments... Aug 7 23:56:03.068437851 (14h16m24.812494719s) [upload=14695]: Got next segment. Inline: true Aug 7 23:56:03.068445058 (14h16m24.812501927s) [upload=14695]: Waiting for error group managing segments... Aug 7 23:56:03.068516175 (14h16m24.812573049s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:03.068523275 (14h16m24.812580137s) [upload=14695]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:03.068527513 (14h16m24.812584488s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:03.370378590 (14h16m25.114435452s) [upload=14695]: Done uploading segments... Aug 7 23:56:03.370401742 (14h16m25.114458656s) [upload=14695]: Upload finished. err: <nil> Aug 7 23:56:41.636734874 (14h17m3.380791742s) [upload=14707]: Starting upload Aug 7 23:56:41.636844337 (14h17m3.380901201s) [upload=14707]: Done starting upload Aug 7 23:56:41.636938361 (14h17m3.380995223s) [upload=14707]: Uploading segments... Aug 7 23:56:41.637004566 (14h17m3.381061436s) [upload=14707]: Got next segment. Inline: true Aug 7 23:56:41.637011720 (14h17m3.381068581s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:56:41.637073380 (14h17m3.381130242s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:41.637079684 (14h17m3.381136546s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:41.637083892 (14h17m3.381140753s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:41.369074470 (14h18m3.113131318s) [upload=14707]: Done uploading segments... Aug 7 23:57:41.369096313 (14h18m3.11315316s) [upload=14707]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:41.785857243 (14h18m3.529914104s) [upload=14726]: Starting upload Aug 7 23:57:41.785973295 (14h18m3.530030155s) [upload=14726]: Done starting upload Aug 7 23:57:41.786053418 (14h18m3.530110268s) [upload=14726]: Uploading segments... Aug 7 23:57:41.786115407 (14h18m3.530172273s) [upload=14726]: Got next segment. Inline: true Aug 7 23:57:41.786121268 (14h18m3.530178127s) [upload=14726]: Waiting for error group managing segments... Aug 7 23:57:41.786162041 (14h18m3.530218901s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:41.786167247 (14h18m3.530224108s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:41.786170583 (14h18m3.530227443s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.518215592 (14h19m3.262272452s) [upload=14726]: Done uploading segments... Aug 7 23:58:41.518237426 (14h19m3.262294275s) [upload=14726]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:49.080161447 (14h19m10.824218292s) [upload=14741]: Starting upload Aug 7 23:58:49.080236560 (14h19m10.824293407s) [upload=14741]: Done starting upload Aug 7 23:58:49.080313897 (14h19m10.824370744s) [upload=14741]: Uploading segments... Aug 7 23:58:49.080371825 (14h19m10.824428691s) [upload=14741]: Got next segment. Inline: true Aug 7 23:58:49.080376725 (14h19m10.824433586s) [upload=14741]: Waiting for error group managing segments... Aug 7 23:58:49.080418334 (14h19m10.824475197s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:49.080422560 (14h19m10.824479419s) [upload=14741]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:49.080438323 (14h19m10.824495186s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:49.220444289 (14h19m10.964501139s) [upload=14741]: Done uploading segments... Aug 7 23:58:49.220462660 (14h19m10.96451951s) [upload=14741]: Upload finished. err: <nil> Aug 7 23:59:26.271206280 (14h19m48.015263132s) [upload=14752]: Starting upload Aug 7 23:59:26.271300842 (14h19m48.015357706s) [upload=14752]: Done starting upload Aug 7 23:59:26.271409542 (14h19m48.015466389s) [upload=14752]: Uploading segments... Aug 7 23:59:26.271471969 (14h19m48.015528839s) [upload=14752]: Got next segment. Inline: true Aug 7 23:59:26.271478422 (14h19m48.015535286s) [upload=14752]: Waiting for error group managing segments... Aug 7 23:59:26.271524304 (14h19m48.015581168s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.271531803 (14h19m48.01558867s) [upload=14752]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:26.271535740 (14h19m48.015592604s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.006104454 (14h20m47.750161323s) [upload=14752]: Done uploading segments... Aug 8 00:00:26.006140966 (14h20m47.750197816s) [upload=14752]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:26.408809053 (14h20m48.152865927s) [upload=14766]: Starting upload Aug 8 00:00:26.408886561 (14h20m48.152943434s) [upload=14766]: Done starting upload Aug 8 00:00:26.408975679 (14h20m48.15303253s) [upload=14766]: Uploading segments... Aug 8 00:00:26.409028539 (14h20m48.153085476s) [upload=14766]: Got next segment. Inline: true Aug 8 00:00:26.409034850 (14h20m48.153091712s) [upload=14766]: Waiting for error group managing segments... Aug 8 00:00:26.409085190 (14h20m48.153142042s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:26.409090930 (14h20m48.153147793s) [upload=14766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:26.409094742 (14h20m48.153151605s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.549853950 (14h20m48.2939108s) [upload=14766]: Done uploading segments... Aug 8 00:00:26.549878419 (14h20m48.293935265s) [upload=14766]: Upload finished. err: <nil> Aug 8 00:01:08.274172938 (14h21m30.018229799s) [upload=14780]: Starting upload Aug 8 00:01:08.274248443 (14h21m30.018305302s) [upload=14780]: Done starting upload Aug 8 00:01:08.274326739 (14h21m30.018383585s) [upload=14780]: Uploading segments... Aug 8 00:01:08.274380689 (14h21m30.018437549s) [upload=14780]: Got next segment. Inline: true Aug 8 00:01:08.274385704 (14h21m30.018442565s) [upload=14780]: Waiting for error group managing segments... Aug 8 00:01:08.274437335 (14h21m30.018494182s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:08.274442821 (14h21m30.018499667s) [upload=14780]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:08.274445609 (14h21m30.018502454s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:08.005343335 (14h22m29.749400197s) [upload=14780]: Done uploading segments... Aug 8 00:02:08.005365121 (14h22m29.749421982s) [upload=14780]: Upload finished. err: metaclient: context deadline exceeded Duration: 872.26492ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [X.X.........................................X.XXX.X.X.....X.] duration: 899.485325ms FAIL (2025-08-08 00:03:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 899.521182ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.XX........................................X.XX...XX.X..X..] duration: 2.254717237s FAIL (2025-08-08 00:03:08 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.254754497s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 824.813806ms FAIL (2025-08-08 00:03:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14121]: Done uploading segments... Aug 7 23:22:28.505845197 (13h42m50.249902044s) [upload=14121]: Upload finished. err: <nil> Aug 7 23:23:10.377483605 (13h43m32.121540458s) [upload=14132]: Starting upload Aug 7 23:23:10.377535635 (13h43m32.121592495s) [upload=14132]: Done starting upload Aug 7 23:23:10.377603797 (13h43m32.121660647s) [upload=14132]: Uploading segments... Aug 7 23:23:10.377638935 (13h43m32.121695804s) [upload=14132]: Got next segment. Inline: true Aug 7 23:23:10.377643594 (13h43m32.121700454s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:23:10.377688252 (13h43m32.121745104s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:23:10.377693427 (13h43m32.121750286s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:23:10.377696805 (13h43m32.121753666s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:23:10.562260724 (13h43m32.306317573s) [upload=14132]: Done uploading segments... Aug 7 23:23:10.562279228 (13h43m32.306336078s) [upload=14132]: Upload finished. err: <nil> Aug 7 23:24:07.896404858 (13h44m29.640461721s) [upload=14147]: Starting upload Aug 7 23:24:07.896490216 (13h44m29.640547079s) [upload=14147]: Done starting upload Aug 7 23:24:07.896580488 (13h44m29.640637339s) [upload=14147]: Uploading segments... Aug 7 23:24:07.896655006 (13h44m29.640711874s) [upload=14147]: Got next segment. Inline: true Aug 7 23:24:07.896660179 (13h44m29.640717039s) [upload=14147]: Waiting for error group managing segments... Aug 7 23:24:07.896714840 (13h44m29.6407717s) [upload=14147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:24:07.896719399 (13h44m29.64077626s) [upload=14147]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:24:07.896722726 (13h44m29.640779586s) [upload=14147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:24:08.048898130 (13h44m29.792954991s) [upload=14147]: Done uploading segments... Aug 7 23:24:08.048952526 (13h44m29.793009386s) [upload=14147]: Upload finished. err: <nil> Aug 7 23:24:53.794754379 (13h45m15.53881125s) [upload=14162]: Starting upload Aug 7 23:24:53.794842317 (13h45m15.538899177s) [upload=14162]: Done starting upload Aug 7 23:24:53.794935964 (13h45m15.538992814s) [upload=14162]: Uploading segments... Aug 7 23:24:53.795003752 (13h45m15.539060623s) [upload=14162]: Got next segment. Inline: true Aug 7 23:24:53.795010814 (13h45m15.539067676s) [upload=14162]: Waiting for error group managing segments... Aug 7 23:24:53.795066607 (13h45m15.539123488s) [upload=14162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:24:53.795158449 (13h45m15.5392153s) [upload=14162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:24:53.795163166 (13h45m15.539220026s) [upload=14162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:24:53.946994982 (13h45m15.691051833s) [upload=14162]: Done uploading segments... Aug 7 23:24:53.947013862 (13h45m15.691070711s) [upload=14162]: Upload finished. err: <nil> Aug 7 23:25:40.029856966 (13h46m1.773913836s) [upload=14177]: Starting upload Aug 7 23:25:40.029986938 (13h46m1.774043809s) [upload=14177]: Done starting upload Aug 7 23:25:40.030076796 (13h46m1.774133642s) [upload=14177]: Uploading segments... Aug 7 23:25:40.030132567 (13h46m1.774189438s) [upload=14177]: Got next segment. Inline: true Aug 7 23:25:40.030139885 (13h46m1.774196757s) [upload=14177]: Waiting for error group managing segments... Aug 7 23:25:40.030197588 (13h46m1.774254461s) [upload=14177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:25:40.030204146 (13h46m1.774261016s) [upload=14177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:25:40.030207984 (13h46m1.77426485s) [upload=14177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:25:40.184198742 (13h46m1.928255592s) [upload=14177]: Done uploading segments... Aug 7 23:25:40.184219017 (13h46m1.928275867s) [upload=14177]: Upload finished. err: <nil> Aug 7 23:26:54.625649110 (13h47m16.369705971s) [upload=14198]: Starting upload Aug 7 23:26:54.625738563 (13h47m16.369795424s) [upload=14198]: Done starting upload Aug 7 23:26:54.625842691 (13h47m16.369899539s) [upload=14198]: Uploading segments... Aug 7 23:26:54.626016839 (13h47m16.37007371s) [upload=14198]: Got next segment. Inline: true Aug 7 23:26:54.626052112 (13h47m16.370108973s) [upload=14198]: Waiting for error group managing segments... Aug 7 23:26:54.626129780 (13h47m16.370186645s) [upload=14198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:26:54.626137392 (13h47m16.370194264s) [upload=14198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:26:54.626143182 (13h47m16.370200048s) [upload=14198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:26:54.775804171 (13h47m16.519861021s) [upload=14198]: Done uploading segments... Aug 7 23:26:54.775824249 (13h47m16.519881102s) [upload=14198]: Upload finished. err: <nil> Aug 7 23:27:32.263746490 (13h47m54.007803361s) [upload=14208]: Starting upload Aug 7 23:27:32.263833655 (13h47m54.007890501s) [upload=14208]: Done starting upload Aug 7 23:27:32.263976536 (13h47m54.008033385s) [upload=14208]: Uploading segments... Aug 7 23:27:32.264070926 (13h47m54.008127796s) [upload=14208]: Got next segment. Inline: true Aug 7 23:27:32.264080356 (13h47m54.00813733s) [upload=14208]: Waiting for error group managing segments... Aug 7 23:27:32.264154657 (13h47m54.008211526s) [upload=14208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:27:32.264163587 (13h47m54.008220454s) [upload=14208]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:27:32.264172017 (13h47m54.008228914s) [upload=14208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:27:32.490188559 (13h47m54.23424541s) [upload=14208]: Done uploading segments... Aug 7 23:27:32.490207326 (13h47m54.234264172s) [upload=14208]: Upload finished. err: <nil> Aug 7 23:27:57.927940477 (13h48m19.671997323s) [upload=14217]: Starting upload Aug 7 23:27:57.928016423 (13h48m19.672073283s) [upload=14217]: Done starting upload Aug 7 23:27:57.928101290 (13h48m19.672158142s) [upload=14217]: Uploading segments... Aug 7 23:27:57.928155692 (13h48m19.672212566s) [upload=14217]: Got next segment. Inline: true Aug 7 23:27:57.928162009 (13h48m19.672218869s) [upload=14217]: Waiting for error group managing segments... Aug 7 23:27:57.928213294 (13h48m19.672270145s) [upload=14217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:27:57.928219212 (13h48m19.672276074s) [upload=14217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:27:57.928223521 (13h48m19.672280383s) [upload=14217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:27:58.087544005 (13h48m19.831600874s) [upload=14217]: Done uploading segments... Aug 7 23:27:58.087570226 (13h48m19.831627094s) [upload=14217]: Upload finished. err: <nil> Aug 7 23:29:12.765200935 (13h49m34.509257795s) [upload=14237]: Starting upload Aug 7 23:29:12.765253106 (13h49m34.509309977s) [upload=14237]: Done starting upload Aug 7 23:29:12.765324641 (13h49m34.509381499s) [upload=14237]: Uploading segments... Aug 7 23:29:12.765374187 (13h49m34.509431056s) [upload=14237]: Got next segment. Inline: true Aug 7 23:29:12.765378947 (13h49m34.509435807s) [upload=14237]: Waiting for error group managing segments... Aug 7 23:29:12.765430859 (13h49m34.50948772s) [upload=14237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:29:12.765438106 (13h49m34.50949497s) [upload=14237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:29:12.765443127 (13h49m34.509499988s) [upload=14237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:29:12.926437674 (13h49m34.670494535s) [upload=14237]: Done uploading segments... Aug 7 23:29:12.926457418 (13h49m34.670514268s) [upload=14237]: Upload finished. err: <nil> Aug 7 23:29:42.225760316 (13h50m3.969817177s) [upload=14244]: Starting upload Aug 7 23:29:42.225855063 (13h50m3.969911929s) [upload=14244]: Done starting upload Aug 7 23:29:42.225962489 (13h50m3.970019355s) [upload=14244]: Uploading segments... Aug 7 23:29:42.226078484 (13h50m3.970135331s) [upload=14244]: Got next segment. Inline: true Aug 7 23:29:42.226083996 (13h50m3.970140842s) [upload=14244]: Waiting for error group managing segments... Aug 7 23:29:42.226143067 (13h50m3.970199914s) [upload=14244]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:29:42.226148511 (13h50m3.970205371s) [upload=14244]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:29:42.226153006 (13h50m3.970209865s) [upload=14244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:29:42.401822947 (13h50m4.14587981s) [upload=14244]: Done uploading segments... Aug 7 23:29:42.401847556 (13h50m4.145904405s) [upload=14244]: Upload finished. err: <nil> Aug 7 23:30:31.529840301 (13h50m53.273897165s) [upload=14254]: Starting upload Aug 7 23:30:31.530002531 (13h50m53.274059378s) [upload=14254]: Done starting upload Aug 7 23:30:31.530201544 (13h50m53.274258391s) [upload=14254]: Uploading segments... Aug 7 23:30:31.530298120 (13h50m53.274354985s) [upload=14254]: Got next segment. Inline: true Aug 7 23:30:31.530303457 (13h50m53.274360317s) [upload=14254]: Waiting for error group managing segments... Aug 7 23:30:31.530353721 (13h50m53.274410567s) [upload=14254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:30:31.530358927 (13h50m53.274415788s) [upload=14254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:30:31.530363758 (13h50m53.274420619s) [upload=14254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:30:31.696350967 (13h50m53.440407828s) [upload=14254]: Done uploading segments... Aug 7 23:30:31.696369460 (13h50m53.440426307s) [upload=14254]: Upload finished. err: <nil> Duration: 824.892836ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X...X.XX.......................................XXX..XX..] duration: 3.065705248s FAIL (2025-08-08 00:03:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 807513521 (14h17m32.551570367s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:55.062716703 (14h18m16.806773554s) [seg_pos={0 0}, upload=14717]: Begin upload segment... Aug 7 23:57:55.062772247 (14h18m16.806829106s) [seg_pos={0 0}, upload=14717]: Done begin upload segment. Aug 7 23:57:55.062817614 (14h18m16.806874477s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.062833275 (14h18m16.806890146s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063321348 (14h18m16.807378211s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063625241 (14h18m16.8076821s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.064591461 (14h18m16.808648334s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065249737 (14h18m16.809306608s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065844042 (14h18m16.809900909s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.066738904 (14h18m16.810795771s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067295868 (14h18m16.81135274s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067868039 (14h18m16.811924903s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068378475 (14h18m16.812435346s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068951556 (14h18m16.81300842s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069355853 (14h18m16.813412717s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069707384 (14h18m16.813764245s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070103904 (14h18m16.814160763s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070434322 (14h18m16.814491182s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070775335 (14h18m16.814832199s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.072531247 (14h18m16.816588107s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.642344995 (14h18m17.386401842s) [upload=14717]: Next returned nil segment Aug 7 23:57:55.642365727 (14h18m17.38642259s) [upload=14717]: Waiting for error group managing segments... Aug 7 23:57:55.820297514 (14h18m17.564354366s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.832273438 (14h18m17.576330291s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.845342869 (14h18m17.58939972s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.850001106 (14h18m17.594057955s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874147776 (14h18m17.618204625s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874555951 (14h18m17.6186128s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.890084811 (14h18m17.634141672s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.901998021 (14h18m17.646054868s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.930296357 (14h18m17.674353212s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.008745690 (14h18m17.752802541s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.041830348 (14h18m17.785887195s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.086405192 (14h18m17.830462041s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.092724415 (14h18m17.83678127s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095750120 (14h18m17.839806985s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095784368 (14h18m17.839841231s) [seg_pos={0 0}, upload=14717]: Segment reached optimal threshold of 14 pieces. Aug 7 23:57:56.095960472 (14h18m17.840017334s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096160491 (14h18m17.840217355s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096190692 (14h18m17.840247554s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eu7SDhEXZGS6HeAXFZWfxoci2oPdjtYvZ5Bbhvae2neKTxYKo): protocol: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection Aug 7 23:57:56.096286416 (14h18m17.840343264s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w4v4QV7PxhXTwuxbvTQST2buoWNySrziRYTo9Mb8dpENd9XCX): protocol: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection Aug 7 23:57:56.096300170 (14h18m17.840357019s) [seg_pos={0 0}, upload=14717]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:57:56.096419305 (14h18m17.840476164s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:56.096426206 (14h18m17.84048307s) [upload=14717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:30.094636522 (14h22m51.838693376s) [upload=14717]: Done uploading segments... Aug 8 00:02:30.094667105 (14h22m51.838723955s) [upload=14717]: Upload finished. err: metaclient: context deadline exceeded Duration:* 3.065762229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 11 out of 60 is failed (threshold=3) [XX..XX.X...................................X...XX.X...XX....] duration: 3.15631253s FAIL (2025-08-08 00:03:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 140355 (14h13m13.036197215s) [upload=14641]: Starting upload Aug 7 23:52:51.292216133 (14h13m13.036272985s) [upload=14641]: Done starting upload Aug 7 23:52:51.292307446 (14h13m13.036364307s) [upload=14641]: Uploading segments... Aug 7 23:52:51.292389087 (14h13m13.036445956s) [upload=14641]: Got next segment. Inline: true Aug 7 23:52:51.292394933 (14h13m13.036451793s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:52:51.292456032 (14h13m13.036512883s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:51.292461787 (14h13m13.036518647s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:51.292467676 (14h13m13.036524537s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:51.436915828 (14h13m13.180972678s) [upload=14641]: Done uploading segments... Aug 7 23:52:51.436934458 (14h13m13.18099131s) [upload=14641]: Upload finished. err: <nil> Aug 7 23:53:10.946261153 (14h13m32.690318028s) [upload=14646]: Starting upload Aug 7 23:53:10.946350198 (14h13m32.690407065s) [upload=14646]: Done starting upload Aug 7 23:53:10.946435423 (14h13m32.690492276s) [upload=14646]: Uploading segments... Aug 7 23:53:10.946489255 (14h13m32.690546124s) [upload=14646]: Got next segment. Inline: true Aug 7 23:53:10.946497660 (14h13m32.690554519s) [upload=14646]: Waiting for error group managing segments... Aug 7 23:53:10.946542678 (14h13m32.690599529s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:10.946548267 (14h13m32.690605128s) [upload=14646]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:10.946553792 (14h13m32.690610652s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:14.554651436 (14h13m36.298708309s) [upload=14646]: Done uploading segments... Aug 7 23:53:14.554675183 (14h13m36.298732045s) [upload=14646]: Upload finished. err: <nil> Aug 7 23:53:53.584502029 (14h14m15.328558897s) [upload=14658]: Starting upload Aug 7 23:53:53.584599757 (14h14m15.328656616s) [upload=14658]: Done starting upload Aug 7 23:53:53.584702364 (14h14m15.328759218s) [upload=14658]: Uploading segments... Aug 7 23:53:53.584798610 (14h14m15.32885548s) [upload=14658]: Got next segment. Inline: true Aug 7 23:53:53.584806594 (14h14m15.328863461s) [upload=14658]: Waiting for error group managing segments... Aug 7 23:53:53.584875193 (14h14m15.328932062s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:53.584882324 (14h14m15.328939193s) [upload=14658]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:53.584901657 (14h14m15.328958528s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:23.318821682 (14h14m45.062878542s) [upload=14658]: Done uploading segments... Aug 7 23:54:23.318863033 (14h14m45.062919884s) [upload=14658]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:36.775209359 (14h14m58.519266209s) [upload=14671]: Starting upload Aug 7 23:54:36.775292235 (14h14m58.519349112s) [upload=14671]: Done starting upload Aug 7 23:54:36.775412031 (14h14m58.519468883s) [upload=14671]: Uploading segments... Aug 7 23:54:36.775480767 (14h14m58.519537636s) [upload=14671]: Got next segment. Inline: true Aug 7 23:54:36.775487320 (14h14m58.519544181s) [upload=14671]: Waiting for error group managing segments... Aug 7 23:54:36.775539752 (14h14m58.519596612s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:36.775545893 (14h14m58.519602753s) [upload=14671]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:36.775549619 (14h14m58.519606479s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:36.926276427 (14h14m58.670333277s) [upload=14671]: Done uploading segments... Aug 7 23:54:36.926295206 (14h14m58.670352057s) [upload=14671]: Upload finished. err: <nil> Aug 7 23:54:57.675347843 (14h15m19.419404694s) [upload=14676]: Starting upload Aug 7 23:54:57.675412352 (14h15m19.419469212s) [upload=14676]: Done starting upload Aug 7 23:54:57.675475670 (14h15m19.41953252s) [upload=14676]: Uploading segments... Aug 7 23:54:57.675516336 (14h15m19.419573205s) [upload=14676]: Got next segment. Inline: true Aug 7 23:54:57.675521071 (14h15m19.419577931s) [upload=14676]: Waiting for error group managing segments... Aug 7 23:54:57.675565512 (14h15m19.419622375s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:57.675574255 (14h15m19.419631202s) [upload=14676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:57.675580694 (14h15m19.419637563s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:27.407552485 (14h15m49.151609344s) [upload=14676]: Done uploading segments... Aug 7 23:55:27.407592978 (14h15m49.151649829s) [upload=14676]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:06.851646232 (14h16m28.595703082s) [upload=14696]: Starting upload Aug 7 23:56:06.851717330 (14h16m28.595774191s) [upload=14696]: Done starting upload Aug 7 23:56:06.851786008 (14h16m28.59584286s) [upload=14696]: Uploading segments... Aug 7 23:56:06.851834232 (14h16m28.595891101s) [upload=14696]: Got next segment. Inline: true Aug 7 23:56:06.851839675 (14h16m28.595896535s) [upload=14696]: Waiting for error group managing segments... Aug 7 23:56:06.851896275 (14h16m28.595953125s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:06.851901659 (14h16m28.59595852s) [upload=14696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:06.851905505 (14h16m28.595962364s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:07.006202008 (14h16m28.750258868s) [upload=14696]: Done uploading segments... Aug 7 23:56:07.006222000 (14h16m28.750278861s) [upload=14696]: Upload finished. err: <nil> Aug 7 23:56:59.020763574 (14h17m20.764820421s) [upload=14712]: Starting upload Aug 7 23:56:59.020858955 (14h17m20.764915806s) [upload=14712]: Done starting upload Aug 7 23:56:59.020935444 (14h17m20.764992295s) [upload=14712]: Uploading segments... Aug 7 23:56:59.021007501 (14h17m20.76506437s) [upload=14712]: Got next segment. Inline: true Aug 7 23:56:59.021013631 (14h17m20.765070491s) [upload=14712]: Waiting for error group managing segments... Aug 7 23:56:59.021070471 (14h17m20.76512733s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:59.021076966 (14h17m20.765133821s) [upload=14712]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:59.021081305 (14h17m20.76513816s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:59.154820708 (14h17m20.898877569s) [upload=14712]: Done uploading segments... Aug 7 23:56:59.154839835 (14h17m20.898896685s) [upload=14712]: Upload finished. err: <nil> Aug 7 23:58:10.963677657 (14h18m32.707734508s) [upload=14728]: Starting upload Aug 7 23:58:10.963770332 (14h18m32.70782718s) [upload=14728]: Done starting upload Aug 7 23:58:10.963851725 (14h18m32.707908586s) [upload=14728]: Uploading segments... Aug 7 23:58:10.963901559 (14h18m32.707958431s) [upload=14728]: Got next segment. Inline: true Aug 7 23:58:10.963907663 (14h18m32.707964525s) [upload=14728]: Waiting for error group managing segments... Aug 7 23:58:10.963960167 (14h18m32.708017032s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:10.963966034 (14h18m32.708022896s) [upload=14728]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:10.963969944 (14h18m32.708026806s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.106069715 (14h18m32.850126571s) [upload=14728]: Done uploading segments... Aug 7 23:58:11.106112099 (14h18m32.850168961s) [upload=14728]: Upload finished. err: <nil> Aug 7 23:59:19.078128070 (14h19m40.822184931s) [upload=14749]: Starting upload Aug 7 23:59:19.078217499 (14h19m40.822274359s) [upload=14749]: Done starting upload Aug 7 23:59:19.078291040 (14h19m40.822347895s) [upload=14749]: Uploading segments... Aug 7 23:59:19.078333863 (14h19m40.822390732s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:19.078339134 (14h19m40.822395984s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:19.078391868 (14h19m40.822448717s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:19.078397213 (14h19m40.822454064s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:19.078401054 (14h19m40.822457905s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:48.808547234 (14h20m10.552604081s) [upload=14749]: Done uploading segments... Aug 7 23:59:48.808568563 (14h20m10.552625409s) [upload=14749]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:20.896255037 (14h20m42.640311905s) [upload=14765]: Starting upload Aug 8 00:00:20.896381648 (14h20m42.640438517s) [upload=14765]: Done starting upload Aug 8 00:00:20.896532416 (14h20m42.640589282s) [upload=14765]: Uploading segments... Aug 8 00:00:20.896610121 (14h20m42.640666994s) [upload=14765]: Got next segment. Inline: true Aug 8 00:00:20.896618785 (14h20m42.640675649s) [upload=14765]: Waiting for error group managing segments... Aug 8 00:00:20.896682486 (14h20m42.640739601s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:20.896690306 (14h20m42.640747187s) [upload=14765]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:20.896694946 (14h20m42.640751814s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:50.607106781 (14h21m12.351163663s) [upload=14765]: Done uploading segments... Aug 8 00:00:50.607134684 (14h21m12.351191558s) [upload=14765]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.156343866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.................................X...] duration: 3.084243103s FAIL (2025-08-08 00:02:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after e4bc51e4c3741c8242928f6ea44cd583e1de2244d7996b5999976d42b698bf65 != before 74576f6679446814770053fb5ecf3ed143c5a0d1682c10852d0fdad56f6f62fd ======LOGS====== em_ObjectBegin Aug 8 00:00:04.218463528 (14h20m25.962520394s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:04.422837522 (14h20m26.166894398s) [seg_pos={0 0}, upload=14759]: Begin upload segment... Aug 8 00:00:04.422913763 (14h20m26.166970631s) [seg_pos={0 0}, upload=14759]: Done begin upload segment. Aug 8 00:00:04.422953631 (14h20m26.167010505s) [noise=true , address=64.49.13.138:4490 , piece_id=VWBCADPZQBKS6HWUJZ6YWK7FDXN6Z2TSBJKK4Z7P3SDYS476UG5A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.423659818 (14h20m26.167716937s) [noise=true , address=104.36.82.38:4713 , piece_id=44IRUAVNYKZVR74WG64GQPGH3J337QYLGYWC3BUOZ3VS5NPJL27A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.424122548 (14h20m26.168179422s) [noise=true , address=79.127.201.214:4295 , piece_id=CCDJBXRBUJ3NL4PDRRCTXBYXWYBIPYSNU3T3I6R4CTJWJRZTGGQA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.424515211 (14h20m26.168572087s) [noise=true , address=104.36.82.19:4404 , piece_id=LHQKAL4W24GGRXIZJ672W6CXY3NDMUN6NJVH2SFRHETKJMEHW62Q, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.424927726 (14h20m26.1689846s) [noise=true , address=104.36.82.3:4796 , piece_id=APJB3DPBNQBOFK34LZ5MPOOR45ZXRQSMFWB4SOCHFOJCSDW3GWVQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.425328696 (14h20m26.169385571s) [noise=true , address=64.49.13.139:4470 , piece_id=WOXVGGQF2Q5PM6V3LGUOMMA4ZU7UPVW2KOR25TQ32PXN6OWCGGLA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.425736715 (14h20m26.169793589s) [noise=true , address=104.36.82.17:4335 , piece_id=TB54EDHJ5XMYQPXWQB2NMUECRW5VTECH7H6KOPLCFUJ5SXS7U4OQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.426503554 (14h20m26.170560432s) [noise=true , address=104.36.82.59:4318 , piece_id=EHUOWZNGTRHSK7HXYEYYREX66ES4MPYKEU34EW66PFUPHI7VS3XQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.430790810 (14h20m26.174847683s) [noise=true , address=64.49.13.155:4916 , piece_id=54U5CI5CPB7DR56LS2GCOWNANROABC65PXTMSSVUTMZDQEZJY7CA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.431475312 (14h20m26.175532189s) [noise=true , address=104.36.82.62:4425 , piece_id=BOX4GUARMRXU5IIE64CELJHQZFT6UWG4TFDATZHX5YFCJTFV2N4A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.431922656 (14h20m26.175979526s) [noise=true , address=104.36.82.58:4268 , piece_id=YFPLOYXHUU3SOZJHFEMSTCNBM55AYJ5NONP7GLTDFU3OTZTW4BIQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.432313695 (14h20m26.176370573s) [noise=true , address=104.36.82.55:4221 , piece_id=G3QN6J3Q4ZSDPAESESJMAWK3ICJLFBWPWY76QGCSMTHKYP4SUJ6Q, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.432740279 (14h20m26.17679715s) [noise=true , address=45.91.116.14:4195 , piece_id=OFPVRGPPQRMNV3FV46GHPFWSXSNSMDDM7MZ7MFO5D4SP7NS6UZGA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.433162962 (14h20m26.177219835s) [noise=true , address=104.36.82.44:4885 , piece_id=ZHPCTPQUPDXJDPZQPBFSOD6KYKAVCHXQNXESSKEXNDU3XSL6OY3A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.433650237 (14h20m26.177707112s) [noise=true , address=104.36.82.43:4854 , piece_id=CY3WAFZSTIQ2AV6DYPIOZSOORUBPL3FBZK4CZNWL3SMOVG7DSZGQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.434316480 (14h20m26.178373354s) [noise=true , address=64.49.13.26:4948 , piece_id=TN7TAN4AYCIDUXZHESWSGMPXYDEM3ZN5RIS5HV3OVOALIFLJDQ7A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.443916849 (14h20m26.187973724s) [noise=true , address=104.36.82.61:4393 , piece_id=MIKYUC5I7GASL3XCYWW7QHLYNJIKSLIHAPNF6NMV45ACHHJLTJJQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.444505957 (14h20m26.188562832s) [noise=true , address=45.91.116.6:4982 , piece_id=M3LYDOCIFKY5YHDBN6ZX6XGSUASZP74QRPZX3VFYSWD37QGHFNBA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:05.200540577 (14h20m26.944597434s) [upload=14759]: Next returned nil segment Aug 8 00:00:05.200687222 (14h20m26.944744097s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:05.355754277 (14h20m27.099811131s) [noise=true , address=104.36.82.59:4318 , piece_id=EHUOWZNGTRHSK7HXYEYYREX66ES4MPYKEU34EW66PFUPHI7VS3XQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.358198826 (14h20m27.102255692s) [noise=true , address=104.36.82.58:4268 , piece_id=YFPLOYXHUU3SOZJHFEMSTCNBM55AYJ5NONP7GLTDFU3OTZTW4BIQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.386961372 (14h20m27.13101822s) [noise=true , address=104.36.82.19:4404 , piece_id=LHQKAL4W24GGRXIZJ672W6CXY3NDMUN6NJVH2SFRHETKJMEHW62Q, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.391547939 (14h20m27.135604807s) [noise=true , address=104.36.82.55:4221 , piece_id=G3QN6J3Q4ZSDPAESESJMAWK3ICJLFBWPWY76QGCSMTHKYP4SUJ6Q, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.391724549 (14h20m27.135781419s) [noise=true , address=104.36.82.43:4854 , piece_id=CY3WAFZSTIQ2AV6DYPIOZSOORUBPL3FBZK4CZNWL3SMOVG7DSZGQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.625387364 (14h20m27.369444214s) [noise=true , address=104.36.82.17:4335 , piece_id=TB54EDHJ5XMYQPXWQB2NMUECRW5VTECH7H6KOPLCFUJ5SXS7U4OQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.685379596 (14h20m27.429436466s) [noise=true , address=104.36.82.38:4713 , piece_id=44IRUAVNYKZVR74WG64GQPGH3J337QYLGYWC3BUOZ3VS5NPJL27A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.696642486 (14h20m27.440699351s) [noise=true , address=45.91.116.6:4982 , piece_id=M3LYDOCIFKY5YHDBN6ZX6XGSUASZP74QRPZX3VFYSWD37QGHFNBA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.722664674 (14h20m27.466721544s) [noise=true , address=104.36.82.3:4796 , piece_id=APJB3DPBNQBOFK34LZ5MPOOR45ZXRQSMFWB4SOCHFOJCSDW3GWVQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.733358308 (14h20m27.477415158s) [noise=true , address=104.36.82.62:4425 , piece_id=BOX4GUARMRXU5IIE64CELJHQZFT6UWG4TFDATZHX5YFCJTFV2N4A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.781882083 (14h20m27.525938951s) [noise=true , address=45.91.116.14:4195 , piece_id=OFPVRGPPQRMNV3FV46GHPFWSXSNSMDDM7MZ7MFO5D4SP7NS6UZGA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.908897662 (14h20m27.652954524s) [noise=true , address=79.127.201.214:4295 , piece_id=CCDJBXRBUJ3NL4PDRRCTXBYXWYBIPYSNU3T3I6R4CTJWJRZTGGQA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:06.052322455 (14h20m27.796379308s) [noise=true , address=64.49.13.139:4470 , piece_id=WOXVGGQF2Q5PM6V3LGUOMMA4ZU7UPVW2KOR25TQ32PXN6OWCGGLA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:06.059613546 (14h20m27.803670401s) [noise=true , address=64.49.13.155:4916 , piece_id=54U5CI5CPB7DR56LS2GCOWNANROABC65PXTMSSVUTMZDQEZJY7CA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:06.059650800 (14h20m27.803707652s) [seg_pos={0 0}, upload=14759]: Segment reached optimal threshold of 14 pieces. Aug 8 00:00:06.061105436 (14h20m27.805162307s) [noise=true , address=64.49.13.138:4490 , piece_id=VWBCADPZQBKS6HWUJZ6YWK7FDXN6Z2TSBJKK4Z7P3SDYS476UG5A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 8 00:00:06.061315764 (14h20m27.805372629s) [noise=true , address=64.49.13.26:4948 , piece_id=TN7TAN4AYCIDUXZHESWSGMPXYDEM3ZN5RIS5HV3OVOALIFLJDQ7A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVV4rAAZRd4DcLsakTTyPRNWck7AjBA6StKvsVLH2dCKVkwHB): protocol: expected piece hash; context canceled; EOF Aug 8 00:00:06.061491394 (14h20m27.805548286s) [noise=true , address=104.36.82.44:4885 , piece_id=ZHPCTPQUPDXJDPZQPBFSOD6KYKAVCHXQNXESSKEXNDU3XSL6OY3A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: write tcp 139.59.144.22:35754->104.36.82.44:4885: use of closed network connection; write tcp 139.59.144.22:35754->104.36.82.44:4885: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35754->104.36.82.44:4885: use of closed network connection Aug 8 00:00:06.078661046 (14h20m27.822717918s) [noise=true , address=104.36.82.61:4393 , piece_id=MIKYUC5I7GASL3XCYWW7QHLYNJIKSLIHAPNF6NMV45ACHHJLTJJQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xfCSdYjRRxMaTw47dQmaXzGmMWyV1oCCGx7JdhvGqaaoN14Ff): protocol: write tcp 139.59.144.22:48102->104.36.82.61:4393: use of closed network connection; write tcp 139.59.144.22:48102->104.36.82.61:4393: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48102->104.36.82.61:4393: use of closed network connection Aug 8 00:00:06.078739080 (14h20m27.822795929s) [seg_pos={0 0}, upload=14759]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:00:06.078890768 (14h20m27.82294764s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:00:06.078899977 (14h20m27.82295685s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:06.568180745 (14h20m28.312237613s) [upload=14759]: Done uploading segments... Aug 8 00:00:06.568206007 (14h20m28.312262863s) [upload=14759]: Upload finished. err: <nil> Duration:* 3.084277536s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXX...........X............................XX.X.X...XX.....] duration: 4.083924644s FAIL (2025-08-08 00:02:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 6]: Uploading piece... Aug 7 23:57:10.893816491 (14h17m32.637873354s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.894285524 (14h17m32.638342391s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895348932 (14h17m32.639405787s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895932061 (14h17m32.639988921s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.896545320 (14h17m32.640602184s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897117472 (14h17m32.641174341s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897621325 (14h17m32.641678192s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898101338 (14h17m32.642158204s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898693607 (14h17m32.642750474s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.900512475 (14h17m32.644569335s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.902337804 (14h17m32.646394669s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.904215667 (14h17m32.648272537s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.906034481 (14h17m32.650091365s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.907857426 (14h17m32.651914295s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.908666188 (14h17m32.652723069s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:11.510238622 (14h17m33.254295484s) [upload=14716]: Next returned nil segment Aug 7 23:57:11.510261464 (14h17m33.254318324s) [upload=14716]: Waiting for error group managing segments... Aug 7 23:57:11.657094388 (14h17m33.401151236s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.674408374 (14h17m33.418465234s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.681203586 (14h17m33.425260449s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.682780850 (14h17m33.4268377s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.684042315 (14h17m33.428099176s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.719658547 (14h17m33.463715401s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.730348391 (14h17m33.474405242s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.748264399 (14h17m33.492321249s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.769764028 (14h17m33.513820878s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.774177874 (14h17m33.518234733s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.789864472 (14h17m33.533921326s) [noise=true , address=45.91.116.13:4153 , piece_id=WBFFQT5I6ZR5ZKUJRFULNOJZ455KIJMA5EEATDDOCVFUKRD67NFA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.799519597 (14h17m33.543576447s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800753821 (14h17m33.544810667s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800785983 (14h17m33.544842842s) [seg_pos={0 0}, upload=14716]: Segment reached optimal threshold of 13 pieces. Aug 7 23:57:11.801026052 (14h17m33.545082898s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801098020 (14h17m33.545154867s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801174944 (14h17m33.54523179s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection Aug 7 23:57:11.801184926 (14h17m33.545241773s) [seg_pos={0 0}, upload=14716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:57:11.801257096 (14h17m33.545313956s) [upload=14716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:11.801261009 (14h17m33.545317869s) [upload=14716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:11.942784613 (14h17m33.686841513s) [upload=14716]: Done uploading segments... Aug 7 23:57:11.942815424 (14h17m33.686872279s) [upload=14716]: Upload finished. err: <nil> Aug 7 23:58:20.561350310 (14h18m42.305407184s) [upload=14732]: Starting upload Aug 7 23:58:20.561483402 (14h18m42.30554027s) [upload=14732]: Done starting upload Aug 7 23:58:20.561650297 (14h18m42.305707159s) [upload=14732]: Uploading segments... Aug 7 23:58:20.562196341 (14h18m42.306253212s) [upload=14732]: Got next segment. Inline: false Aug 7 23:58:20.562334166 (14h18m42.306391031s) [upload=14732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:20.562343255 (14h18m42.306400125s) [upload=14732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:50.281195862 (14h19m12.025252712s) [upload=14732]: Next segment err: context deadline exceeded Aug 7 23:58:50.281383705 (14h19m12.025440555s) [upload=14732]: Done uploading segments... Aug 7 23:58:50.281405600 (14h19m12.02546245s) [upload=14732]: Upload finished. err: context deadline exceeded Aug 7 23:58:50.716777496 (14h19m12.460834346s) [upload=14742]: Starting upload Aug 7 23:58:50.716857905 (14h19m12.460914751s) [upload=14742]: Done starting upload Aug 7 23:58:50.716999972 (14h19m12.461056825s) [upload=14742]: Uploading segments... Aug 7 23:58:50.717186124 (14h19m12.461242976s) [upload=14742]: Got next segment. Inline: false Aug 7 23:58:50.717245601 (14h19m12.461302466s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:50.717252185 (14h19m12.461309053s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:20.436529435 (14h19m42.180586305s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:20.436734882 (14h19m42.180791733s) [upload=14742]: Done uploading segments... Aug 7 23:59:20.436756200 (14h19m42.180813054s) [upload=14742]: Upload finished. err: context deadline exceeded Duration: 4.083977939s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [..XX.X..............................................X...X...] duration: 1.076309074s FAIL (2025-08-08 00:02:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.076335171s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX.XX........................................X.XX...XX.X..X.] duration: 25.726616554s FAIL (2025-08-08 00:02:39 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 25.726680986s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.990767696s FAIL (2025-08-08 00:02:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.990801617s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXXX...........X............................XX.X.X...XX....] duration: 12.968382471s FAIL (2025-08-08 00:02:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 6]: Uploading piece... Aug 7 23:57:10.893816491 (14h17m32.637873354s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.894285524 (14h17m32.638342391s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895348932 (14h17m32.639405787s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895932061 (14h17m32.639988921s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.896545320 (14h17m32.640602184s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897117472 (14h17m32.641174341s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897621325 (14h17m32.641678192s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898101338 (14h17m32.642158204s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898693607 (14h17m32.642750474s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.900512475 (14h17m32.644569335s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.902337804 (14h17m32.646394669s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.904215667 (14h17m32.648272537s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.906034481 (14h17m32.650091365s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.907857426 (14h17m32.651914295s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.908666188 (14h17m32.652723069s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:11.510238622 (14h17m33.254295484s) [upload=14716]: Next returned nil segment Aug 7 23:57:11.510261464 (14h17m33.254318324s) [upload=14716]: Waiting for error group managing segments... Aug 7 23:57:11.657094388 (14h17m33.401151236s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.674408374 (14h17m33.418465234s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.681203586 (14h17m33.425260449s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.682780850 (14h17m33.4268377s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.684042315 (14h17m33.428099176s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.719658547 (14h17m33.463715401s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.730348391 (14h17m33.474405242s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.748264399 (14h17m33.492321249s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.769764028 (14h17m33.513820878s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.774177874 (14h17m33.518234733s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.789864472 (14h17m33.533921326s) [noise=true , address=45.91.116.13:4153 , piece_id=WBFFQT5I6ZR5ZKUJRFULNOJZ455KIJMA5EEATDDOCVFUKRD67NFA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.799519597 (14h17m33.543576447s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800753821 (14h17m33.544810667s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800785983 (14h17m33.544842842s) [seg_pos={0 0}, upload=14716]: Segment reached optimal threshold of 13 pieces. Aug 7 23:57:11.801026052 (14h17m33.545082898s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801098020 (14h17m33.545154867s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801174944 (14h17m33.54523179s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection Aug 7 23:57:11.801184926 (14h17m33.545241773s) [seg_pos={0 0}, upload=14716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:57:11.801257096 (14h17m33.545313956s) [upload=14716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:11.801261009 (14h17m33.545317869s) [upload=14716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:11.942784613 (14h17m33.686841513s) [upload=14716]: Done uploading segments... Aug 7 23:57:11.942815424 (14h17m33.686872279s) [upload=14716]: Upload finished. err: <nil> Aug 7 23:58:20.561350310 (14h18m42.305407184s) [upload=14732]: Starting upload Aug 7 23:58:20.561483402 (14h18m42.30554027s) [upload=14732]: Done starting upload Aug 7 23:58:20.561650297 (14h18m42.305707159s) [upload=14732]: Uploading segments... Aug 7 23:58:20.562196341 (14h18m42.306253212s) [upload=14732]: Got next segment. Inline: false Aug 7 23:58:20.562334166 (14h18m42.306391031s) [upload=14732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:20.562343255 (14h18m42.306400125s) [upload=14732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:50.281195862 (14h19m12.025252712s) [upload=14732]: Next segment err: context deadline exceeded Aug 7 23:58:50.281383705 (14h19m12.025440555s) [upload=14732]: Done uploading segments... Aug 7 23:58:50.281405600 (14h19m12.02546245s) [upload=14732]: Upload finished. err: context deadline exceeded Aug 7 23:58:50.716777496 (14h19m12.460834346s) [upload=14742]: Starting upload Aug 7 23:58:50.716857905 (14h19m12.460914751s) [upload=14742]: Done starting upload Aug 7 23:58:50.716999972 (14h19m12.461056825s) [upload=14742]: Uploading segments... Aug 7 23:58:50.717186124 (14h19m12.461242976s) [upload=14742]: Got next segment. Inline: false Aug 7 23:58:50.717245601 (14h19m12.461302466s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:50.717252185 (14h19m12.461309053s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:20.436529435 (14h19m42.180586305s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:20.436734882 (14h19m42.180791733s) [upload=14742]: Done uploading segments... Aug 7 23:59:20.436756200 (14h19m42.180813054s) [upload=14742]: Upload finished. err: context deadline exceeded Duration: 12.968413925s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [XX.X............................................XXX.X.X.....] duration: 4.711179427s FAIL (2025-08-08 00:02:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.711531877s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X...X.XX.......................................XXX..XX.] duration: 2.837751647s FAIL (2025-08-08 00:02:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 807513521 (14h17m32.551570367s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:55.062716703 (14h18m16.806773554s) [seg_pos={0 0}, upload=14717]: Begin upload segment... Aug 7 23:57:55.062772247 (14h18m16.806829106s) [seg_pos={0 0}, upload=14717]: Done begin upload segment. Aug 7 23:57:55.062817614 (14h18m16.806874477s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.062833275 (14h18m16.806890146s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063321348 (14h18m16.807378211s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063625241 (14h18m16.8076821s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.064591461 (14h18m16.808648334s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065249737 (14h18m16.809306608s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065844042 (14h18m16.809900909s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.066738904 (14h18m16.810795771s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067295868 (14h18m16.81135274s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067868039 (14h18m16.811924903s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068378475 (14h18m16.812435346s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068951556 (14h18m16.81300842s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069355853 (14h18m16.813412717s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069707384 (14h18m16.813764245s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070103904 (14h18m16.814160763s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070434322 (14h18m16.814491182s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070775335 (14h18m16.814832199s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.072531247 (14h18m16.816588107s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.642344995 (14h18m17.386401842s) [upload=14717]: Next returned nil segment Aug 7 23:57:55.642365727 (14h18m17.38642259s) [upload=14717]: Waiting for error group managing segments... Aug 7 23:57:55.820297514 (14h18m17.564354366s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.832273438 (14h18m17.576330291s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.845342869 (14h18m17.58939972s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.850001106 (14h18m17.594057955s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874147776 (14h18m17.618204625s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874555951 (14h18m17.6186128s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.890084811 (14h18m17.634141672s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.901998021 (14h18m17.646054868s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.930296357 (14h18m17.674353212s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.008745690 (14h18m17.752802541s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.041830348 (14h18m17.785887195s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.086405192 (14h18m17.830462041s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.092724415 (14h18m17.83678127s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095750120 (14h18m17.839806985s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095784368 (14h18m17.839841231s) [seg_pos={0 0}, upload=14717]: Segment reached optimal threshold of 14 pieces. Aug 7 23:57:56.095960472 (14h18m17.840017334s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096160491 (14h18m17.840217355s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096190692 (14h18m17.840247554s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eu7SDhEXZGS6HeAXFZWfxoci2oPdjtYvZ5Bbhvae2neKTxYKo): protocol: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection Aug 7 23:57:56.096286416 (14h18m17.840343264s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w4v4QV7PxhXTwuxbvTQST2buoWNySrziRYTo9Mb8dpENd9XCX): protocol: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection Aug 7 23:57:56.096300170 (14h18m17.840357019s) [seg_pos={0 0}, upload=14717]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:57:56.096419305 (14h18m17.840476164s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:56.096426206 (14h18m17.84048307s) [upload=14717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:30.094636522 (14h22m51.838693376s) [upload=14717]: Done uploading segments... Aug 8 00:02:30.094667105 (14h22m51.838723955s) [upload=14717]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.837827918s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [XX.X.........................................X.XXX.X.X.....X] duration: 30.001604173s FAIL (2025-08-08 00:02:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001631018s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X...X.XX.......................................XXX..XX] duration: 5m19.557233769s FAIL (2025-08-08 00:02:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 807513521 (14h17m32.551570367s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:55.062716703 (14h18m16.806773554s) [seg_pos={0 0}, upload=14717]: Begin upload segment... Aug 7 23:57:55.062772247 (14h18m16.806829106s) [seg_pos={0 0}, upload=14717]: Done begin upload segment. Aug 7 23:57:55.062817614 (14h18m16.806874477s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.062833275 (14h18m16.806890146s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063321348 (14h18m16.807378211s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063625241 (14h18m16.8076821s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.064591461 (14h18m16.808648334s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065249737 (14h18m16.809306608s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065844042 (14h18m16.809900909s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.066738904 (14h18m16.810795771s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067295868 (14h18m16.81135274s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067868039 (14h18m16.811924903s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068378475 (14h18m16.812435346s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068951556 (14h18m16.81300842s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069355853 (14h18m16.813412717s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069707384 (14h18m16.813764245s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070103904 (14h18m16.814160763s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070434322 (14h18m16.814491182s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070775335 (14h18m16.814832199s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.072531247 (14h18m16.816588107s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.642344995 (14h18m17.386401842s) [upload=14717]: Next returned nil segment Aug 7 23:57:55.642365727 (14h18m17.38642259s) [upload=14717]: Waiting for error group managing segments... Aug 7 23:57:55.820297514 (14h18m17.564354366s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.832273438 (14h18m17.576330291s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.845342869 (14h18m17.58939972s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.850001106 (14h18m17.594057955s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874147776 (14h18m17.618204625s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874555951 (14h18m17.6186128s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.890084811 (14h18m17.634141672s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.901998021 (14h18m17.646054868s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.930296357 (14h18m17.674353212s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.008745690 (14h18m17.752802541s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.041830348 (14h18m17.785887195s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.086405192 (14h18m17.830462041s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.092724415 (14h18m17.83678127s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095750120 (14h18m17.839806985s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095784368 (14h18m17.839841231s) [seg_pos={0 0}, upload=14717]: Segment reached optimal threshold of 14 pieces. Aug 7 23:57:56.095960472 (14h18m17.840017334s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096160491 (14h18m17.840217355s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096190692 (14h18m17.840247554s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eu7SDhEXZGS6HeAXFZWfxoci2oPdjtYvZ5Bbhvae2neKTxYKo): protocol: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection Aug 7 23:57:56.096286416 (14h18m17.840343264s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w4v4QV7PxhXTwuxbvTQST2buoWNySrziRYTo9Mb8dpENd9XCX): protocol: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection Aug 7 23:57:56.096300170 (14h18m17.840357019s) [seg_pos={0 0}, upload=14717]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:57:56.096419305 (14h18m17.840476164s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:56.096426206 (14h18m17.84048307s) [upload=14717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:30.094636522 (14h22m51.838693376s) [upload=14717]: Done uploading segments... Aug 8 00:02:30.094667105 (14h22m51.838723955s) [upload=14717]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5m19.557261651s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 105.100667ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-08 00:02:28 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 105.140001ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 321.807692ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-08 00:02:23 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 321.841355ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [XXX..XX.X...................................X...XX.X...XX...] duration: 869.886391ms FAIL (2025-08-08 00:02:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 140355 (14h13m13.036197215s) [upload=14641]: Starting upload Aug 7 23:52:51.292216133 (14h13m13.036272985s) [upload=14641]: Done starting upload Aug 7 23:52:51.292307446 (14h13m13.036364307s) [upload=14641]: Uploading segments... Aug 7 23:52:51.292389087 (14h13m13.036445956s) [upload=14641]: Got next segment. Inline: true Aug 7 23:52:51.292394933 (14h13m13.036451793s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:52:51.292456032 (14h13m13.036512883s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:51.292461787 (14h13m13.036518647s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:51.292467676 (14h13m13.036524537s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:51.436915828 (14h13m13.180972678s) [upload=14641]: Done uploading segments... Aug 7 23:52:51.436934458 (14h13m13.18099131s) [upload=14641]: Upload finished. err: <nil> Aug 7 23:53:10.946261153 (14h13m32.690318028s) [upload=14646]: Starting upload Aug 7 23:53:10.946350198 (14h13m32.690407065s) [upload=14646]: Done starting upload Aug 7 23:53:10.946435423 (14h13m32.690492276s) [upload=14646]: Uploading segments... Aug 7 23:53:10.946489255 (14h13m32.690546124s) [upload=14646]: Got next segment. Inline: true Aug 7 23:53:10.946497660 (14h13m32.690554519s) [upload=14646]: Waiting for error group managing segments... Aug 7 23:53:10.946542678 (14h13m32.690599529s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:10.946548267 (14h13m32.690605128s) [upload=14646]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:10.946553792 (14h13m32.690610652s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:14.554651436 (14h13m36.298708309s) [upload=14646]: Done uploading segments... Aug 7 23:53:14.554675183 (14h13m36.298732045s) [upload=14646]: Upload finished. err: <nil> Aug 7 23:53:53.584502029 (14h14m15.328558897s) [upload=14658]: Starting upload Aug 7 23:53:53.584599757 (14h14m15.328656616s) [upload=14658]: Done starting upload Aug 7 23:53:53.584702364 (14h14m15.328759218s) [upload=14658]: Uploading segments... Aug 7 23:53:53.584798610 (14h14m15.32885548s) [upload=14658]: Got next segment. Inline: true Aug 7 23:53:53.584806594 (14h14m15.328863461s) [upload=14658]: Waiting for error group managing segments... Aug 7 23:53:53.584875193 (14h14m15.328932062s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:53.584882324 (14h14m15.328939193s) [upload=14658]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:53.584901657 (14h14m15.328958528s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:23.318821682 (14h14m45.062878542s) [upload=14658]: Done uploading segments... Aug 7 23:54:23.318863033 (14h14m45.062919884s) [upload=14658]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:36.775209359 (14h14m58.519266209s) [upload=14671]: Starting upload Aug 7 23:54:36.775292235 (14h14m58.519349112s) [upload=14671]: Done starting upload Aug 7 23:54:36.775412031 (14h14m58.519468883s) [upload=14671]: Uploading segments... Aug 7 23:54:36.775480767 (14h14m58.519537636s) [upload=14671]: Got next segment. Inline: true Aug 7 23:54:36.775487320 (14h14m58.519544181s) [upload=14671]: Waiting for error group managing segments... Aug 7 23:54:36.775539752 (14h14m58.519596612s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:36.775545893 (14h14m58.519602753s) [upload=14671]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:36.775549619 (14h14m58.519606479s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:36.926276427 (14h14m58.670333277s) [upload=14671]: Done uploading segments... Aug 7 23:54:36.926295206 (14h14m58.670352057s) [upload=14671]: Upload finished. err: <nil> Aug 7 23:54:57.675347843 (14h15m19.419404694s) [upload=14676]: Starting upload Aug 7 23:54:57.675412352 (14h15m19.419469212s) [upload=14676]: Done starting upload Aug 7 23:54:57.675475670 (14h15m19.41953252s) [upload=14676]: Uploading segments... Aug 7 23:54:57.675516336 (14h15m19.419573205s) [upload=14676]: Got next segment. Inline: true Aug 7 23:54:57.675521071 (14h15m19.419577931s) [upload=14676]: Waiting for error group managing segments... Aug 7 23:54:57.675565512 (14h15m19.419622375s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:57.675574255 (14h15m19.419631202s) [upload=14676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:57.675580694 (14h15m19.419637563s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:27.407552485 (14h15m49.151609344s) [upload=14676]: Done uploading segments... Aug 7 23:55:27.407592978 (14h15m49.151649829s) [upload=14676]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:06.851646232 (14h16m28.595703082s) [upload=14696]: Starting upload Aug 7 23:56:06.851717330 (14h16m28.595774191s) [upload=14696]: Done starting upload Aug 7 23:56:06.851786008 (14h16m28.59584286s) [upload=14696]: Uploading segments... Aug 7 23:56:06.851834232 (14h16m28.595891101s) [upload=14696]: Got next segment. Inline: true Aug 7 23:56:06.851839675 (14h16m28.595896535s) [upload=14696]: Waiting for error group managing segments... Aug 7 23:56:06.851896275 (14h16m28.595953125s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:06.851901659 (14h16m28.59595852s) [upload=14696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:06.851905505 (14h16m28.595962364s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:07.006202008 (14h16m28.750258868s) [upload=14696]: Done uploading segments... Aug 7 23:56:07.006222000 (14h16m28.750278861s) [upload=14696]: Upload finished. err: <nil> Aug 7 23:56:59.020763574 (14h17m20.764820421s) [upload=14712]: Starting upload Aug 7 23:56:59.020858955 (14h17m20.764915806s) [upload=14712]: Done starting upload Aug 7 23:56:59.020935444 (14h17m20.764992295s) [upload=14712]: Uploading segments... Aug 7 23:56:59.021007501 (14h17m20.76506437s) [upload=14712]: Got next segment. Inline: true Aug 7 23:56:59.021013631 (14h17m20.765070491s) [upload=14712]: Waiting for error group managing segments... Aug 7 23:56:59.021070471 (14h17m20.76512733s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:59.021076966 (14h17m20.765133821s) [upload=14712]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:59.021081305 (14h17m20.76513816s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:59.154820708 (14h17m20.898877569s) [upload=14712]: Done uploading segments... Aug 7 23:56:59.154839835 (14h17m20.898896685s) [upload=14712]: Upload finished. err: <nil> Aug 7 23:58:10.963677657 (14h18m32.707734508s) [upload=14728]: Starting upload Aug 7 23:58:10.963770332 (14h18m32.70782718s) [upload=14728]: Done starting upload Aug 7 23:58:10.963851725 (14h18m32.707908586s) [upload=14728]: Uploading segments... Aug 7 23:58:10.963901559 (14h18m32.707958431s) [upload=14728]: Got next segment. Inline: true Aug 7 23:58:10.963907663 (14h18m32.707964525s) [upload=14728]: Waiting for error group managing segments... Aug 7 23:58:10.963960167 (14h18m32.708017032s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:10.963966034 (14h18m32.708022896s) [upload=14728]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:10.963969944 (14h18m32.708026806s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.106069715 (14h18m32.850126571s) [upload=14728]: Done uploading segments... Aug 7 23:58:11.106112099 (14h18m32.850168961s) [upload=14728]: Upload finished. err: <nil> Aug 7 23:59:19.078128070 (14h19m40.822184931s) [upload=14749]: Starting upload Aug 7 23:59:19.078217499 (14h19m40.822274359s) [upload=14749]: Done starting upload Aug 7 23:59:19.078291040 (14h19m40.822347895s) [upload=14749]: Uploading segments... Aug 7 23:59:19.078333863 (14h19m40.822390732s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:19.078339134 (14h19m40.822395984s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:19.078391868 (14h19m40.822448717s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:19.078397213 (14h19m40.822454064s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:19.078401054 (14h19m40.822457905s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:48.808547234 (14h20m10.552604081s) [upload=14749]: Done uploading segments... Aug 7 23:59:48.808568563 (14h20m10.552625409s) [upload=14749]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:20.896255037 (14h20m42.640311905s) [upload=14765]: Starting upload Aug 8 00:00:20.896381648 (14h20m42.640438517s) [upload=14765]: Done starting upload Aug 8 00:00:20.896532416 (14h20m42.640589282s) [upload=14765]: Uploading segments... Aug 8 00:00:20.896610121 (14h20m42.640666994s) [upload=14765]: Got next segment. Inline: true Aug 8 00:00:20.896618785 (14h20m42.640675649s) [upload=14765]: Waiting for error group managing segments... Aug 8 00:00:20.896682486 (14h20m42.640739601s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:20.896690306 (14h20m42.640747187s) [upload=14765]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:20.896694946 (14h20m42.640751814s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:50.607106781 (14h21m12.351163663s) [upload=14765]: Done uploading segments... Aug 8 00:00:50.607134684 (14h21m12.351191558s) [upload=14765]: Upload finished. err: metaclient: context deadline exceeded Duration: 869.962781ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXX.XX........................................X.XX...XX.X..X] duration: 30.000723292s FAIL (2025-08-08 00:02:11 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.00083481s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [XXX.X............................................XXX.X.X....] duration: 6.255671136s FAIL (2025-08-08 00:02:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.255705551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [X................................................XX..XX.X.X.] duration: 891.075971ms FAIL (2025-08-08 00:02:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0]: Starting upload Aug 7 23:53:17.153036870 (14h13m38.89709373s) [upload=14650]: Done starting upload Aug 7 23:53:17.153154678 (14h13m38.897211544s) [upload=14650]: Uploading segments... Aug 7 23:53:17.153233089 (14h13m38.897289957s) [upload=14650]: Got next segment. Inline: true Aug 7 23:53:17.153240256 (14h13m38.897297124s) [upload=14650]: Waiting for error group managing segments... Aug 7 23:53:17.153318892 (14h13m38.897375757s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:17.153325816 (14h13m38.89738268s) [upload=14650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:17.153330453 (14h13m38.897387314s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:16.875757982 (14h14m38.619814845s) [upload=14650]: Done uploading segments... Aug 7 23:54:16.875781849 (14h14m38.619838701s) [upload=14650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:24.158448119 (14h14m45.90250498s) [upload=14667]: Starting upload Aug 7 23:54:24.158525946 (14h14m45.902582806s) [upload=14667]: Done starting upload Aug 7 23:54:24.158598972 (14h14m45.902655824s) [upload=14667]: Uploading segments... Aug 7 23:54:24.158654148 (14h14m45.902711017s) [upload=14667]: Got next segment. Inline: true Aug 7 23:54:24.158659995 (14h14m45.902716856s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:54:24.158714896 (14h14m45.902771747s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:24.158720357 (14h14m45.902777209s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:24.158724105 (14h14m45.902780965s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:23.888400190 (14h15m45.63245704s) [upload=14667]: Done uploading segments... Aug 7 23:55:23.888419087 (14h15m45.632475938s) [upload=14667]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:24.297146309 (14h15m46.041203174s) [upload=14687]: Starting upload Aug 7 23:55:24.297232144 (14h15m46.04128901s) [upload=14687]: Done starting upload Aug 7 23:55:24.297315772 (14h15m46.041372619s) [upload=14687]: Uploading segments... Aug 7 23:55:24.297396238 (14h15m46.041453151s) [upload=14687]: Got next segment. Inline: true Aug 7 23:55:24.297403729 (14h15m46.04146059s) [upload=14687]: Waiting for error group managing segments... Aug 7 23:55:24.297444116 (14h15m46.041500976s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:24.297450971 (14h15m46.041507831s) [upload=14687]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:24.297453883 (14h15m46.041510749s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:24.439126204 (14h15m46.183183055s) [upload=14687]: Done uploading segments... Aug 7 23:55:24.439141121 (14h15m46.183197972s) [upload=14687]: Upload finished. err: <nil> Aug 7 23:56:03.068174127 (14h16m24.812230977s) [upload=14695]: Starting upload Aug 7 23:56:03.068258755 (14h16m24.812315617s) [upload=14695]: Done starting upload Aug 7 23:56:03.068368961 (14h16m24.812425812s) [upload=14695]: Uploading segments... Aug 7 23:56:03.068437851 (14h16m24.812494719s) [upload=14695]: Got next segment. Inline: true Aug 7 23:56:03.068445058 (14h16m24.812501927s) [upload=14695]: Waiting for error group managing segments... Aug 7 23:56:03.068516175 (14h16m24.812573049s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:03.068523275 (14h16m24.812580137s) [upload=14695]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:03.068527513 (14h16m24.812584488s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:03.370378590 (14h16m25.114435452s) [upload=14695]: Done uploading segments... Aug 7 23:56:03.370401742 (14h16m25.114458656s) [upload=14695]: Upload finished. err: <nil> Aug 7 23:56:41.636734874 (14h17m3.380791742s) [upload=14707]: Starting upload Aug 7 23:56:41.636844337 (14h17m3.380901201s) [upload=14707]: Done starting upload Aug 7 23:56:41.636938361 (14h17m3.380995223s) [upload=14707]: Uploading segments... Aug 7 23:56:41.637004566 (14h17m3.381061436s) [upload=14707]: Got next segment. Inline: true Aug 7 23:56:41.637011720 (14h17m3.381068581s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:56:41.637073380 (14h17m3.381130242s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:41.637079684 (14h17m3.381136546s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:41.637083892 (14h17m3.381140753s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:41.369074470 (14h18m3.113131318s) [upload=14707]: Done uploading segments... Aug 7 23:57:41.369096313 (14h18m3.11315316s) [upload=14707]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:41.785857243 (14h18m3.529914104s) [upload=14726]: Starting upload Aug 7 23:57:41.785973295 (14h18m3.530030155s) [upload=14726]: Done starting upload Aug 7 23:57:41.786053418 (14h18m3.530110268s) [upload=14726]: Uploading segments... Aug 7 23:57:41.786115407 (14h18m3.530172273s) [upload=14726]: Got next segment. Inline: true Aug 7 23:57:41.786121268 (14h18m3.530178127s) [upload=14726]: Waiting for error group managing segments... Aug 7 23:57:41.786162041 (14h18m3.530218901s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:41.786167247 (14h18m3.530224108s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:41.786170583 (14h18m3.530227443s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.518215592 (14h19m3.262272452s) [upload=14726]: Done uploading segments... Aug 7 23:58:41.518237426 (14h19m3.262294275s) [upload=14726]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:49.080161447 (14h19m10.824218292s) [upload=14741]: Starting upload Aug 7 23:58:49.080236560 (14h19m10.824293407s) [upload=14741]: Done starting upload Aug 7 23:58:49.080313897 (14h19m10.824370744s) [upload=14741]: Uploading segments... Aug 7 23:58:49.080371825 (14h19m10.824428691s) [upload=14741]: Got next segment. Inline: true Aug 7 23:58:49.080376725 (14h19m10.824433586s) [upload=14741]: Waiting for error group managing segments... Aug 7 23:58:49.080418334 (14h19m10.824475197s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:49.080422560 (14h19m10.824479419s) [upload=14741]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:49.080438323 (14h19m10.824495186s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:49.220444289 (14h19m10.964501139s) [upload=14741]: Done uploading segments... Aug 7 23:58:49.220462660 (14h19m10.96451951s) [upload=14741]: Upload finished. err: <nil> Aug 7 23:59:26.271206280 (14h19m48.015263132s) [upload=14752]: Starting upload Aug 7 23:59:26.271300842 (14h19m48.015357706s) [upload=14752]: Done starting upload Aug 7 23:59:26.271409542 (14h19m48.015466389s) [upload=14752]: Uploading segments... Aug 7 23:59:26.271471969 (14h19m48.015528839s) [upload=14752]: Got next segment. Inline: true Aug 7 23:59:26.271478422 (14h19m48.015535286s) [upload=14752]: Waiting for error group managing segments... Aug 7 23:59:26.271524304 (14h19m48.015581168s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.271531803 (14h19m48.01558867s) [upload=14752]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:26.271535740 (14h19m48.015592604s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.006104454 (14h20m47.750161323s) [upload=14752]: Done uploading segments... Aug 8 00:00:26.006140966 (14h20m47.750197816s) [upload=14752]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:26.408809053 (14h20m48.152865927s) [upload=14766]: Starting upload Aug 8 00:00:26.408886561 (14h20m48.152943434s) [upload=14766]: Done starting upload Aug 8 00:00:26.408975679 (14h20m48.15303253s) [upload=14766]: Uploading segments... Aug 8 00:00:26.409028539 (14h20m48.153085476s) [upload=14766]: Got next segment. Inline: true Aug 8 00:00:26.409034850 (14h20m48.153091712s) [upload=14766]: Waiting for error group managing segments... Aug 8 00:00:26.409085190 (14h20m48.153142042s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:26.409090930 (14h20m48.153147793s) [upload=14766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:26.409094742 (14h20m48.153151605s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.549853950 (14h20m48.2939108s) [upload=14766]: Done uploading segments... Aug 8 00:00:26.549878419 (14h20m48.293935265s) [upload=14766]: Upload finished. err: <nil> Aug 8 00:01:08.274172938 (14h21m30.018229799s) [upload=14780]: Starting upload Aug 8 00:01:08.274248443 (14h21m30.018305302s) [upload=14780]: Done starting upload Aug 8 00:01:08.274326739 (14h21m30.018383585s) [upload=14780]: Uploading segments... Aug 8 00:01:08.274380689 (14h21m30.018437549s) [upload=14780]: Got next segment. Inline: true Aug 8 00:01:08.274385704 (14h21m30.018442565s) [upload=14780]: Waiting for error group managing segments... Aug 8 00:01:08.274437335 (14h21m30.018494182s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:08.274442821 (14h21m30.018499667s) [upload=14780]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:08.274445609 (14h21m30.018502454s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:08.005343335 (14h22m29.749400197s) [upload=14780]: Done uploading segments... Aug 8 00:02:08.005365121 (14h22m29.749421982s) [upload=14780]: Upload finished. err: metaclient: context deadline exceeded Duration: 891.098517ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [XX................................................XX..XX.X.X] duration: 1m0.001427758s FAIL (2025-08-08 00:02:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0]: Starting upload Aug 7 23:53:17.153036870 (14h13m38.89709373s) [upload=14650]: Done starting upload Aug 7 23:53:17.153154678 (14h13m38.897211544s) [upload=14650]: Uploading segments... Aug 7 23:53:17.153233089 (14h13m38.897289957s) [upload=14650]: Got next segment. Inline: true Aug 7 23:53:17.153240256 (14h13m38.897297124s) [upload=14650]: Waiting for error group managing segments... Aug 7 23:53:17.153318892 (14h13m38.897375757s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:17.153325816 (14h13m38.89738268s) [upload=14650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:17.153330453 (14h13m38.897387314s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:16.875757982 (14h14m38.619814845s) [upload=14650]: Done uploading segments... Aug 7 23:54:16.875781849 (14h14m38.619838701s) [upload=14650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:24.158448119 (14h14m45.90250498s) [upload=14667]: Starting upload Aug 7 23:54:24.158525946 (14h14m45.902582806s) [upload=14667]: Done starting upload Aug 7 23:54:24.158598972 (14h14m45.902655824s) [upload=14667]: Uploading segments... Aug 7 23:54:24.158654148 (14h14m45.902711017s) [upload=14667]: Got next segment. Inline: true Aug 7 23:54:24.158659995 (14h14m45.902716856s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:54:24.158714896 (14h14m45.902771747s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:24.158720357 (14h14m45.902777209s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:24.158724105 (14h14m45.902780965s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:23.888400190 (14h15m45.63245704s) [upload=14667]: Done uploading segments... Aug 7 23:55:23.888419087 (14h15m45.632475938s) [upload=14667]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:24.297146309 (14h15m46.041203174s) [upload=14687]: Starting upload Aug 7 23:55:24.297232144 (14h15m46.04128901s) [upload=14687]: Done starting upload Aug 7 23:55:24.297315772 (14h15m46.041372619s) [upload=14687]: Uploading segments... Aug 7 23:55:24.297396238 (14h15m46.041453151s) [upload=14687]: Got next segment. Inline: true Aug 7 23:55:24.297403729 (14h15m46.04146059s) [upload=14687]: Waiting for error group managing segments... Aug 7 23:55:24.297444116 (14h15m46.041500976s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:24.297450971 (14h15m46.041507831s) [upload=14687]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:24.297453883 (14h15m46.041510749s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:24.439126204 (14h15m46.183183055s) [upload=14687]: Done uploading segments... Aug 7 23:55:24.439141121 (14h15m46.183197972s) [upload=14687]: Upload finished. err: <nil> Aug 7 23:56:03.068174127 (14h16m24.812230977s) [upload=14695]: Starting upload Aug 7 23:56:03.068258755 (14h16m24.812315617s) [upload=14695]: Done starting upload Aug 7 23:56:03.068368961 (14h16m24.812425812s) [upload=14695]: Uploading segments... Aug 7 23:56:03.068437851 (14h16m24.812494719s) [upload=14695]: Got next segment. Inline: true Aug 7 23:56:03.068445058 (14h16m24.812501927s) [upload=14695]: Waiting for error group managing segments... Aug 7 23:56:03.068516175 (14h16m24.812573049s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:03.068523275 (14h16m24.812580137s) [upload=14695]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:03.068527513 (14h16m24.812584488s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:03.370378590 (14h16m25.114435452s) [upload=14695]: Done uploading segments... Aug 7 23:56:03.370401742 (14h16m25.114458656s) [upload=14695]: Upload finished. err: <nil> Aug 7 23:56:41.636734874 (14h17m3.380791742s) [upload=14707]: Starting upload Aug 7 23:56:41.636844337 (14h17m3.380901201s) [upload=14707]: Done starting upload Aug 7 23:56:41.636938361 (14h17m3.380995223s) [upload=14707]: Uploading segments... Aug 7 23:56:41.637004566 (14h17m3.381061436s) [upload=14707]: Got next segment. Inline: true Aug 7 23:56:41.637011720 (14h17m3.381068581s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:56:41.637073380 (14h17m3.381130242s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:41.637079684 (14h17m3.381136546s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:41.637083892 (14h17m3.381140753s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:41.369074470 (14h18m3.113131318s) [upload=14707]: Done uploading segments... Aug 7 23:57:41.369096313 (14h18m3.11315316s) [upload=14707]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:41.785857243 (14h18m3.529914104s) [upload=14726]: Starting upload Aug 7 23:57:41.785973295 (14h18m3.530030155s) [upload=14726]: Done starting upload Aug 7 23:57:41.786053418 (14h18m3.530110268s) [upload=14726]: Uploading segments... Aug 7 23:57:41.786115407 (14h18m3.530172273s) [upload=14726]: Got next segment. Inline: true Aug 7 23:57:41.786121268 (14h18m3.530178127s) [upload=14726]: Waiting for error group managing segments... Aug 7 23:57:41.786162041 (14h18m3.530218901s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:41.786167247 (14h18m3.530224108s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:41.786170583 (14h18m3.530227443s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.518215592 (14h19m3.262272452s) [upload=14726]: Done uploading segments... Aug 7 23:58:41.518237426 (14h19m3.262294275s) [upload=14726]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:49.080161447 (14h19m10.824218292s) [upload=14741]: Starting upload Aug 7 23:58:49.080236560 (14h19m10.824293407s) [upload=14741]: Done starting upload Aug 7 23:58:49.080313897 (14h19m10.824370744s) [upload=14741]: Uploading segments... Aug 7 23:58:49.080371825 (14h19m10.824428691s) [upload=14741]: Got next segment. Inline: true Aug 7 23:58:49.080376725 (14h19m10.824433586s) [upload=14741]: Waiting for error group managing segments... Aug 7 23:58:49.080418334 (14h19m10.824475197s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:49.080422560 (14h19m10.824479419s) [upload=14741]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:49.080438323 (14h19m10.824495186s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:49.220444289 (14h19m10.964501139s) [upload=14741]: Done uploading segments... Aug 7 23:58:49.220462660 (14h19m10.96451951s) [upload=14741]: Upload finished. err: <nil> Aug 7 23:59:26.271206280 (14h19m48.015263132s) [upload=14752]: Starting upload Aug 7 23:59:26.271300842 (14h19m48.015357706s) [upload=14752]: Done starting upload Aug 7 23:59:26.271409542 (14h19m48.015466389s) [upload=14752]: Uploading segments... Aug 7 23:59:26.271471969 (14h19m48.015528839s) [upload=14752]: Got next segment. Inline: true Aug 7 23:59:26.271478422 (14h19m48.015535286s) [upload=14752]: Waiting for error group managing segments... Aug 7 23:59:26.271524304 (14h19m48.015581168s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.271531803 (14h19m48.01558867s) [upload=14752]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:26.271535740 (14h19m48.015592604s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.006104454 (14h20m47.750161323s) [upload=14752]: Done uploading segments... Aug 8 00:00:26.006140966 (14h20m47.750197816s) [upload=14752]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:26.408809053 (14h20m48.152865927s) [upload=14766]: Starting upload Aug 8 00:00:26.408886561 (14h20m48.152943434s) [upload=14766]: Done starting upload Aug 8 00:00:26.408975679 (14h20m48.15303253s) [upload=14766]: Uploading segments... Aug 8 00:00:26.409028539 (14h20m48.153085476s) [upload=14766]: Got next segment. Inline: true Aug 8 00:00:26.409034850 (14h20m48.153091712s) [upload=14766]: Waiting for error group managing segments... Aug 8 00:00:26.409085190 (14h20m48.153142042s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:26.409090930 (14h20m48.153147793s) [upload=14766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:26.409094742 (14h20m48.153151605s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.549853950 (14h20m48.2939108s) [upload=14766]: Done uploading segments... Aug 8 00:00:26.549878419 (14h20m48.293935265s) [upload=14766]: Upload finished. err: <nil> Aug 8 00:01:08.274172938 (14h21m30.018229799s) [upload=14780]: Starting upload Aug 8 00:01:08.274248443 (14h21m30.018305302s) [upload=14780]: Done starting upload Aug 8 00:01:08.274326739 (14h21m30.018383585s) [upload=14780]: Uploading segments... Aug 8 00:01:08.274380689 (14h21m30.018437549s) [upload=14780]: Got next segment. Inline: true Aug 8 00:01:08.274385704 (14h21m30.018442565s) [upload=14780]: Waiting for error group managing segments... Aug 8 00:01:08.274437335 (14h21m30.018494182s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:08.274442821 (14h21m30.018499667s) [upload=14780]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:08.274445609 (14h21m30.018502454s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:08.005343335 (14h22m29.749400197s) [upload=14780]: Done uploading segments... Aug 8 00:02:08.005365121 (14h22m29.749421982s) [upload=14780]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.00145923s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 292.82946ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-08 00:02:08 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 292.857353ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [...XX.X..............................................X...X..] duration: 1.176581182s FAIL (2025-08-08 00:02:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.176603135s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [X...XX.X..............................................X...X.] duration: 15.954631383s FAIL (2025-08-08 00:02:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.954652382s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.582682173s FAIL (2025-08-08 00:01:59 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14121]: Done uploading segments... Aug 7 23:22:28.505845197 (13h42m50.249902044s) [upload=14121]: Upload finished. err: <nil> Aug 7 23:23:10.377483605 (13h43m32.121540458s) [upload=14132]: Starting upload Aug 7 23:23:10.377535635 (13h43m32.121592495s) [upload=14132]: Done starting upload Aug 7 23:23:10.377603797 (13h43m32.121660647s) [upload=14132]: Uploading segments... Aug 7 23:23:10.377638935 (13h43m32.121695804s) [upload=14132]: Got next segment. Inline: true Aug 7 23:23:10.377643594 (13h43m32.121700454s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:23:10.377688252 (13h43m32.121745104s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:23:10.377693427 (13h43m32.121750286s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:23:10.377696805 (13h43m32.121753666s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:23:10.562260724 (13h43m32.306317573s) [upload=14132]: Done uploading segments... Aug 7 23:23:10.562279228 (13h43m32.306336078s) [upload=14132]: Upload finished. err: <nil> Aug 7 23:24:07.896404858 (13h44m29.640461721s) [upload=14147]: Starting upload Aug 7 23:24:07.896490216 (13h44m29.640547079s) [upload=14147]: Done starting upload Aug 7 23:24:07.896580488 (13h44m29.640637339s) [upload=14147]: Uploading segments... Aug 7 23:24:07.896655006 (13h44m29.640711874s) [upload=14147]: Got next segment. Inline: true Aug 7 23:24:07.896660179 (13h44m29.640717039s) [upload=14147]: Waiting for error group managing segments... Aug 7 23:24:07.896714840 (13h44m29.6407717s) [upload=14147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:24:07.896719399 (13h44m29.64077626s) [upload=14147]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:24:07.896722726 (13h44m29.640779586s) [upload=14147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:24:08.048898130 (13h44m29.792954991s) [upload=14147]: Done uploading segments... Aug 7 23:24:08.048952526 (13h44m29.793009386s) [upload=14147]: Upload finished. err: <nil> Aug 7 23:24:53.794754379 (13h45m15.53881125s) [upload=14162]: Starting upload Aug 7 23:24:53.794842317 (13h45m15.538899177s) [upload=14162]: Done starting upload Aug 7 23:24:53.794935964 (13h45m15.538992814s) [upload=14162]: Uploading segments... Aug 7 23:24:53.795003752 (13h45m15.539060623s) [upload=14162]: Got next segment. Inline: true Aug 7 23:24:53.795010814 (13h45m15.539067676s) [upload=14162]: Waiting for error group managing segments... Aug 7 23:24:53.795066607 (13h45m15.539123488s) [upload=14162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:24:53.795158449 (13h45m15.5392153s) [upload=14162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:24:53.795163166 (13h45m15.539220026s) [upload=14162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:24:53.946994982 (13h45m15.691051833s) [upload=14162]: Done uploading segments... Aug 7 23:24:53.947013862 (13h45m15.691070711s) [upload=14162]: Upload finished. err: <nil> Aug 7 23:25:40.029856966 (13h46m1.773913836s) [upload=14177]: Starting upload Aug 7 23:25:40.029986938 (13h46m1.774043809s) [upload=14177]: Done starting upload Aug 7 23:25:40.030076796 (13h46m1.774133642s) [upload=14177]: Uploading segments... Aug 7 23:25:40.030132567 (13h46m1.774189438s) [upload=14177]: Got next segment. Inline: true Aug 7 23:25:40.030139885 (13h46m1.774196757s) [upload=14177]: Waiting for error group managing segments... Aug 7 23:25:40.030197588 (13h46m1.774254461s) [upload=14177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:25:40.030204146 (13h46m1.774261016s) [upload=14177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:25:40.030207984 (13h46m1.77426485s) [upload=14177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:25:40.184198742 (13h46m1.928255592s) [upload=14177]: Done uploading segments... Aug 7 23:25:40.184219017 (13h46m1.928275867s) [upload=14177]: Upload finished. err: <nil> Aug 7 23:26:54.625649110 (13h47m16.369705971s) [upload=14198]: Starting upload Aug 7 23:26:54.625738563 (13h47m16.369795424s) [upload=14198]: Done starting upload Aug 7 23:26:54.625842691 (13h47m16.369899539s) [upload=14198]: Uploading segments... Aug 7 23:26:54.626016839 (13h47m16.37007371s) [upload=14198]: Got next segment. Inline: true Aug 7 23:26:54.626052112 (13h47m16.370108973s) [upload=14198]: Waiting for error group managing segments... Aug 7 23:26:54.626129780 (13h47m16.370186645s) [upload=14198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:26:54.626137392 (13h47m16.370194264s) [upload=14198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:26:54.626143182 (13h47m16.370200048s) [upload=14198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:26:54.775804171 (13h47m16.519861021s) [upload=14198]: Done uploading segments... Aug 7 23:26:54.775824249 (13h47m16.519881102s) [upload=14198]: Upload finished. err: <nil> Aug 7 23:27:32.263746490 (13h47m54.007803361s) [upload=14208]: Starting upload Aug 7 23:27:32.263833655 (13h47m54.007890501s) [upload=14208]: Done starting upload Aug 7 23:27:32.263976536 (13h47m54.008033385s) [upload=14208]: Uploading segments... Aug 7 23:27:32.264070926 (13h47m54.008127796s) [upload=14208]: Got next segment. Inline: true Aug 7 23:27:32.264080356 (13h47m54.00813733s) [upload=14208]: Waiting for error group managing segments... Aug 7 23:27:32.264154657 (13h47m54.008211526s) [upload=14208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:27:32.264163587 (13h47m54.008220454s) [upload=14208]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:27:32.264172017 (13h47m54.008228914s) [upload=14208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:27:32.490188559 (13h47m54.23424541s) [upload=14208]: Done uploading segments... Aug 7 23:27:32.490207326 (13h47m54.234264172s) [upload=14208]: Upload finished. err: <nil> Aug 7 23:27:57.927940477 (13h48m19.671997323s) [upload=14217]: Starting upload Aug 7 23:27:57.928016423 (13h48m19.672073283s) [upload=14217]: Done starting upload Aug 7 23:27:57.928101290 (13h48m19.672158142s) [upload=14217]: Uploading segments... Aug 7 23:27:57.928155692 (13h48m19.672212566s) [upload=14217]: Got next segment. Inline: true Aug 7 23:27:57.928162009 (13h48m19.672218869s) [upload=14217]: Waiting for error group managing segments... Aug 7 23:27:57.928213294 (13h48m19.672270145s) [upload=14217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:27:57.928219212 (13h48m19.672276074s) [upload=14217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:27:57.928223521 (13h48m19.672280383s) [upload=14217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:27:58.087544005 (13h48m19.831600874s) [upload=14217]: Done uploading segments... Aug 7 23:27:58.087570226 (13h48m19.831627094s) [upload=14217]: Upload finished. err: <nil> Aug 7 23:29:12.765200935 (13h49m34.509257795s) [upload=14237]: Starting upload Aug 7 23:29:12.765253106 (13h49m34.509309977s) [upload=14237]: Done starting upload Aug 7 23:29:12.765324641 (13h49m34.509381499s) [upload=14237]: Uploading segments... Aug 7 23:29:12.765374187 (13h49m34.509431056s) [upload=14237]: Got next segment. Inline: true Aug 7 23:29:12.765378947 (13h49m34.509435807s) [upload=14237]: Waiting for error group managing segments... Aug 7 23:29:12.765430859 (13h49m34.50948772s) [upload=14237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:29:12.765438106 (13h49m34.50949497s) [upload=14237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:29:12.765443127 (13h49m34.509499988s) [upload=14237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:29:12.926437674 (13h49m34.670494535s) [upload=14237]: Done uploading segments... Aug 7 23:29:12.926457418 (13h49m34.670514268s) [upload=14237]: Upload finished. err: <nil> Aug 7 23:29:42.225760316 (13h50m3.969817177s) [upload=14244]: Starting upload Aug 7 23:29:42.225855063 (13h50m3.969911929s) [upload=14244]: Done starting upload Aug 7 23:29:42.225962489 (13h50m3.970019355s) [upload=14244]: Uploading segments... Aug 7 23:29:42.226078484 (13h50m3.970135331s) [upload=14244]: Got next segment. Inline: true Aug 7 23:29:42.226083996 (13h50m3.970140842s) [upload=14244]: Waiting for error group managing segments... Aug 7 23:29:42.226143067 (13h50m3.970199914s) [upload=14244]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:29:42.226148511 (13h50m3.970205371s) [upload=14244]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:29:42.226153006 (13h50m3.970209865s) [upload=14244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:29:42.401822947 (13h50m4.14587981s) [upload=14244]: Done uploading segments... Aug 7 23:29:42.401847556 (13h50m4.145904405s) [upload=14244]: Upload finished. err: <nil> Aug 7 23:30:31.529840301 (13h50m53.273897165s) [upload=14254]: Starting upload Aug 7 23:30:31.530002531 (13h50m53.274059378s) [upload=14254]: Done starting upload Aug 7 23:30:31.530201544 (13h50m53.274258391s) [upload=14254]: Uploading segments... Aug 7 23:30:31.530298120 (13h50m53.274354985s) [upload=14254]: Got next segment. Inline: true Aug 7 23:30:31.530303457 (13h50m53.274360317s) [upload=14254]: Waiting for error group managing segments... Aug 7 23:30:31.530353721 (13h50m53.274410567s) [upload=14254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:30:31.530358927 (13h50m53.274415788s) [upload=14254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:30:31.530363758 (13h50m53.274420619s) [upload=14254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:30:31.696350967 (13h50m53.440407828s) [upload=14254]: Done uploading segments... Aug 7 23:30:31.696369460 (13h50m53.440426307s) [upload=14254]: Upload finished. err: <nil> Duration: 1.582718833s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXXXXX...........X............................XX.X.X...XX...] duration: 2.544442442s FAIL (2025-08-08 00:01:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 6]: Uploading piece... Aug 7 23:57:10.893816491 (14h17m32.637873354s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.894285524 (14h17m32.638342391s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895348932 (14h17m32.639405787s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895932061 (14h17m32.639988921s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.896545320 (14h17m32.640602184s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897117472 (14h17m32.641174341s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897621325 (14h17m32.641678192s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898101338 (14h17m32.642158204s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898693607 (14h17m32.642750474s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.900512475 (14h17m32.644569335s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.902337804 (14h17m32.646394669s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.904215667 (14h17m32.648272537s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.906034481 (14h17m32.650091365s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.907857426 (14h17m32.651914295s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.908666188 (14h17m32.652723069s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:11.510238622 (14h17m33.254295484s) [upload=14716]: Next returned nil segment Aug 7 23:57:11.510261464 (14h17m33.254318324s) [upload=14716]: Waiting for error group managing segments... Aug 7 23:57:11.657094388 (14h17m33.401151236s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.674408374 (14h17m33.418465234s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.681203586 (14h17m33.425260449s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.682780850 (14h17m33.4268377s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.684042315 (14h17m33.428099176s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.719658547 (14h17m33.463715401s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.730348391 (14h17m33.474405242s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.748264399 (14h17m33.492321249s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.769764028 (14h17m33.513820878s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.774177874 (14h17m33.518234733s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.789864472 (14h17m33.533921326s) [noise=true , address=45.91.116.13:4153 , piece_id=WBFFQT5I6ZR5ZKUJRFULNOJZ455KIJMA5EEATDDOCVFUKRD67NFA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.799519597 (14h17m33.543576447s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800753821 (14h17m33.544810667s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800785983 (14h17m33.544842842s) [seg_pos={0 0}, upload=14716]: Segment reached optimal threshold of 13 pieces. Aug 7 23:57:11.801026052 (14h17m33.545082898s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801098020 (14h17m33.545154867s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801174944 (14h17m33.54523179s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection Aug 7 23:57:11.801184926 (14h17m33.545241773s) [seg_pos={0 0}, upload=14716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:57:11.801257096 (14h17m33.545313956s) [upload=14716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:11.801261009 (14h17m33.545317869s) [upload=14716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:11.942784613 (14h17m33.686841513s) [upload=14716]: Done uploading segments... Aug 7 23:57:11.942815424 (14h17m33.686872279s) [upload=14716]: Upload finished. err: <nil> Aug 7 23:58:20.561350310 (14h18m42.305407184s) [upload=14732]: Starting upload Aug 7 23:58:20.561483402 (14h18m42.30554027s) [upload=14732]: Done starting upload Aug 7 23:58:20.561650297 (14h18m42.305707159s) [upload=14732]: Uploading segments... Aug 7 23:58:20.562196341 (14h18m42.306253212s) [upload=14732]: Got next segment. Inline: false Aug 7 23:58:20.562334166 (14h18m42.306391031s) [upload=14732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:20.562343255 (14h18m42.306400125s) [upload=14732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:50.281195862 (14h19m12.025252712s) [upload=14732]: Next segment err: context deadline exceeded Aug 7 23:58:50.281383705 (14h19m12.025440555s) [upload=14732]: Done uploading segments... Aug 7 23:58:50.281405600 (14h19m12.02546245s) [upload=14732]: Upload finished. err: context deadline exceeded Aug 7 23:58:50.716777496 (14h19m12.460834346s) [upload=14742]: Starting upload Aug 7 23:58:50.716857905 (14h19m12.460914751s) [upload=14742]: Done starting upload Aug 7 23:58:50.716999972 (14h19m12.461056825s) [upload=14742]: Uploading segments... Aug 7 23:58:50.717186124 (14h19m12.461242976s) [upload=14742]: Got next segment. Inline: false Aug 7 23:58:50.717245601 (14h19m12.461302466s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:50.717252185 (14h19m12.461309053s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:20.436529435 (14h19m42.180586305s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:20.436734882 (14h19m42.180791733s) [upload=14742]: Done uploading segments... Aug 7 23:59:20.436756200 (14h19m42.180813054s) [upload=14742]: Upload finished. err: context deadline exceeded Duration: 2.544473817s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 13 out of 60 is failed (threshold=3) [XXXX..XX.X...................................X...XX.X...XX..] duration: 870.761135ms FAIL (2025-08-08 00:01:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 140355 (14h13m13.036197215s) [upload=14641]: Starting upload Aug 7 23:52:51.292216133 (14h13m13.036272985s) [upload=14641]: Done starting upload Aug 7 23:52:51.292307446 (14h13m13.036364307s) [upload=14641]: Uploading segments... Aug 7 23:52:51.292389087 (14h13m13.036445956s) [upload=14641]: Got next segment. Inline: true Aug 7 23:52:51.292394933 (14h13m13.036451793s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:52:51.292456032 (14h13m13.036512883s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:51.292461787 (14h13m13.036518647s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:51.292467676 (14h13m13.036524537s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:51.436915828 (14h13m13.180972678s) [upload=14641]: Done uploading segments... Aug 7 23:52:51.436934458 (14h13m13.18099131s) [upload=14641]: Upload finished. err: <nil> Aug 7 23:53:10.946261153 (14h13m32.690318028s) [upload=14646]: Starting upload Aug 7 23:53:10.946350198 (14h13m32.690407065s) [upload=14646]: Done starting upload Aug 7 23:53:10.946435423 (14h13m32.690492276s) [upload=14646]: Uploading segments... Aug 7 23:53:10.946489255 (14h13m32.690546124s) [upload=14646]: Got next segment. Inline: true Aug 7 23:53:10.946497660 (14h13m32.690554519s) [upload=14646]: Waiting for error group managing segments... Aug 7 23:53:10.946542678 (14h13m32.690599529s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:10.946548267 (14h13m32.690605128s) [upload=14646]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:10.946553792 (14h13m32.690610652s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:14.554651436 (14h13m36.298708309s) [upload=14646]: Done uploading segments... Aug 7 23:53:14.554675183 (14h13m36.298732045s) [upload=14646]: Upload finished. err: <nil> Aug 7 23:53:53.584502029 (14h14m15.328558897s) [upload=14658]: Starting upload Aug 7 23:53:53.584599757 (14h14m15.328656616s) [upload=14658]: Done starting upload Aug 7 23:53:53.584702364 (14h14m15.328759218s) [upload=14658]: Uploading segments... Aug 7 23:53:53.584798610 (14h14m15.32885548s) [upload=14658]: Got next segment. Inline: true Aug 7 23:53:53.584806594 (14h14m15.328863461s) [upload=14658]: Waiting for error group managing segments... Aug 7 23:53:53.584875193 (14h14m15.328932062s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:53.584882324 (14h14m15.328939193s) [upload=14658]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:53.584901657 (14h14m15.328958528s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:23.318821682 (14h14m45.062878542s) [upload=14658]: Done uploading segments... Aug 7 23:54:23.318863033 (14h14m45.062919884s) [upload=14658]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:36.775209359 (14h14m58.519266209s) [upload=14671]: Starting upload Aug 7 23:54:36.775292235 (14h14m58.519349112s) [upload=14671]: Done starting upload Aug 7 23:54:36.775412031 (14h14m58.519468883s) [upload=14671]: Uploading segments... Aug 7 23:54:36.775480767 (14h14m58.519537636s) [upload=14671]: Got next segment. Inline: true Aug 7 23:54:36.775487320 (14h14m58.519544181s) [upload=14671]: Waiting for error group managing segments... Aug 7 23:54:36.775539752 (14h14m58.519596612s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:36.775545893 (14h14m58.519602753s) [upload=14671]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:36.775549619 (14h14m58.519606479s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:36.926276427 (14h14m58.670333277s) [upload=14671]: Done uploading segments... Aug 7 23:54:36.926295206 (14h14m58.670352057s) [upload=14671]: Upload finished. err: <nil> Aug 7 23:54:57.675347843 (14h15m19.419404694s) [upload=14676]: Starting upload Aug 7 23:54:57.675412352 (14h15m19.419469212s) [upload=14676]: Done starting upload Aug 7 23:54:57.675475670 (14h15m19.41953252s) [upload=14676]: Uploading segments... Aug 7 23:54:57.675516336 (14h15m19.419573205s) [upload=14676]: Got next segment. Inline: true Aug 7 23:54:57.675521071 (14h15m19.419577931s) [upload=14676]: Waiting for error group managing segments... Aug 7 23:54:57.675565512 (14h15m19.419622375s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:57.675574255 (14h15m19.419631202s) [upload=14676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:57.675580694 (14h15m19.419637563s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:27.407552485 (14h15m49.151609344s) [upload=14676]: Done uploading segments... Aug 7 23:55:27.407592978 (14h15m49.151649829s) [upload=14676]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:06.851646232 (14h16m28.595703082s) [upload=14696]: Starting upload Aug 7 23:56:06.851717330 (14h16m28.595774191s) [upload=14696]: Done starting upload Aug 7 23:56:06.851786008 (14h16m28.59584286s) [upload=14696]: Uploading segments... Aug 7 23:56:06.851834232 (14h16m28.595891101s) [upload=14696]: Got next segment. Inline: true Aug 7 23:56:06.851839675 (14h16m28.595896535s) [upload=14696]: Waiting for error group managing segments... Aug 7 23:56:06.851896275 (14h16m28.595953125s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:06.851901659 (14h16m28.59595852s) [upload=14696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:06.851905505 (14h16m28.595962364s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:07.006202008 (14h16m28.750258868s) [upload=14696]: Done uploading segments... Aug 7 23:56:07.006222000 (14h16m28.750278861s) [upload=14696]: Upload finished. err: <nil> Aug 7 23:56:59.020763574 (14h17m20.764820421s) [upload=14712]: Starting upload Aug 7 23:56:59.020858955 (14h17m20.764915806s) [upload=14712]: Done starting upload Aug 7 23:56:59.020935444 (14h17m20.764992295s) [upload=14712]: Uploading segments... Aug 7 23:56:59.021007501 (14h17m20.76506437s) [upload=14712]: Got next segment. Inline: true Aug 7 23:56:59.021013631 (14h17m20.765070491s) [upload=14712]: Waiting for error group managing segments... Aug 7 23:56:59.021070471 (14h17m20.76512733s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:59.021076966 (14h17m20.765133821s) [upload=14712]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:59.021081305 (14h17m20.76513816s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:59.154820708 (14h17m20.898877569s) [upload=14712]: Done uploading segments... Aug 7 23:56:59.154839835 (14h17m20.898896685s) [upload=14712]: Upload finished. err: <nil> Aug 7 23:58:10.963677657 (14h18m32.707734508s) [upload=14728]: Starting upload Aug 7 23:58:10.963770332 (14h18m32.70782718s) [upload=14728]: Done starting upload Aug 7 23:58:10.963851725 (14h18m32.707908586s) [upload=14728]: Uploading segments... Aug 7 23:58:10.963901559 (14h18m32.707958431s) [upload=14728]: Got next segment. Inline: true Aug 7 23:58:10.963907663 (14h18m32.707964525s) [upload=14728]: Waiting for error group managing segments... Aug 7 23:58:10.963960167 (14h18m32.708017032s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:10.963966034 (14h18m32.708022896s) [upload=14728]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:10.963969944 (14h18m32.708026806s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.106069715 (14h18m32.850126571s) [upload=14728]: Done uploading segments... Aug 7 23:58:11.106112099 (14h18m32.850168961s) [upload=14728]: Upload finished. err: <nil> Aug 7 23:59:19.078128070 (14h19m40.822184931s) [upload=14749]: Starting upload Aug 7 23:59:19.078217499 (14h19m40.822274359s) [upload=14749]: Done starting upload Aug 7 23:59:19.078291040 (14h19m40.822347895s) [upload=14749]: Uploading segments... Aug 7 23:59:19.078333863 (14h19m40.822390732s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:19.078339134 (14h19m40.822395984s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:19.078391868 (14h19m40.822448717s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:19.078397213 (14h19m40.822454064s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:19.078401054 (14h19m40.822457905s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:48.808547234 (14h20m10.552604081s) [upload=14749]: Done uploading segments... Aug 7 23:59:48.808568563 (14h20m10.552625409s) [upload=14749]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:20.896255037 (14h20m42.640311905s) [upload=14765]: Starting upload Aug 8 00:00:20.896381648 (14h20m42.640438517s) [upload=14765]: Done starting upload Aug 8 00:00:20.896532416 (14h20m42.640589282s) [upload=14765]: Uploading segments... Aug 8 00:00:20.896610121 (14h20m42.640666994s) [upload=14765]: Got next segment. Inline: true Aug 8 00:00:20.896618785 (14h20m42.640675649s) [upload=14765]: Waiting for error group managing segments... Aug 8 00:00:20.896682486 (14h20m42.640739601s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:20.896690306 (14h20m42.640747187s) [upload=14765]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:20.896694946 (14h20m42.640751814s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:50.607106781 (14h21m12.351163663s) [upload=14765]: Done uploading segments... Aug 8 00:00:50.607134684 (14h21m12.351191558s) [upload=14765]: Upload finished. err: metaclient: context deadline exceeded Duration: 870.792789ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [XX...XX.X..............................................X...X] duration: 30.00131386s FAIL (2025-08-08 00:01:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001394658s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.................................X..] duration: 2.85120342s FAIL (2025-08-08 00:01:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after e4bc51e4c3741c8242928f6ea44cd583e1de2244d7996b5999976d42b698bf65 != before 74576f6679446814770053fb5ecf3ed143c5a0d1682c10852d0fdad56f6f62fd ======LOGS====== em_ObjectBegin Aug 8 00:00:04.218463528 (14h20m25.962520394s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:04.422837522 (14h20m26.166894398s) [seg_pos={0 0}, upload=14759]: Begin upload segment... Aug 8 00:00:04.422913763 (14h20m26.166970631s) [seg_pos={0 0}, upload=14759]: Done begin upload segment. Aug 8 00:00:04.422953631 (14h20m26.167010505s) [noise=true , address=64.49.13.138:4490 , piece_id=VWBCADPZQBKS6HWUJZ6YWK7FDXN6Z2TSBJKK4Z7P3SDYS476UG5A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.423659818 (14h20m26.167716937s) [noise=true , address=104.36.82.38:4713 , piece_id=44IRUAVNYKZVR74WG64GQPGH3J337QYLGYWC3BUOZ3VS5NPJL27A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.424122548 (14h20m26.168179422s) [noise=true , address=79.127.201.214:4295 , piece_id=CCDJBXRBUJ3NL4PDRRCTXBYXWYBIPYSNU3T3I6R4CTJWJRZTGGQA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.424515211 (14h20m26.168572087s) [noise=true , address=104.36.82.19:4404 , piece_id=LHQKAL4W24GGRXIZJ672W6CXY3NDMUN6NJVH2SFRHETKJMEHW62Q, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.424927726 (14h20m26.1689846s) [noise=true , address=104.36.82.3:4796 , piece_id=APJB3DPBNQBOFK34LZ5MPOOR45ZXRQSMFWB4SOCHFOJCSDW3GWVQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.425328696 (14h20m26.169385571s) [noise=true , address=64.49.13.139:4470 , piece_id=WOXVGGQF2Q5PM6V3LGUOMMA4ZU7UPVW2KOR25TQ32PXN6OWCGGLA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.425736715 (14h20m26.169793589s) [noise=true , address=104.36.82.17:4335 , piece_id=TB54EDHJ5XMYQPXWQB2NMUECRW5VTECH7H6KOPLCFUJ5SXS7U4OQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.426503554 (14h20m26.170560432s) [noise=true , address=104.36.82.59:4318 , piece_id=EHUOWZNGTRHSK7HXYEYYREX66ES4MPYKEU34EW66PFUPHI7VS3XQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.430790810 (14h20m26.174847683s) [noise=true , address=64.49.13.155:4916 , piece_id=54U5CI5CPB7DR56LS2GCOWNANROABC65PXTMSSVUTMZDQEZJY7CA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.431475312 (14h20m26.175532189s) [noise=true , address=104.36.82.62:4425 , piece_id=BOX4GUARMRXU5IIE64CELJHQZFT6UWG4TFDATZHX5YFCJTFV2N4A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.431922656 (14h20m26.175979526s) [noise=true , address=104.36.82.58:4268 , piece_id=YFPLOYXHUU3SOZJHFEMSTCNBM55AYJ5NONP7GLTDFU3OTZTW4BIQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.432313695 (14h20m26.176370573s) [noise=true , address=104.36.82.55:4221 , piece_id=G3QN6J3Q4ZSDPAESESJMAWK3ICJLFBWPWY76QGCSMTHKYP4SUJ6Q, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.432740279 (14h20m26.17679715s) [noise=true , address=45.91.116.14:4195 , piece_id=OFPVRGPPQRMNV3FV46GHPFWSXSNSMDDM7MZ7MFO5D4SP7NS6UZGA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.433162962 (14h20m26.177219835s) [noise=true , address=104.36.82.44:4885 , piece_id=ZHPCTPQUPDXJDPZQPBFSOD6KYKAVCHXQNXESSKEXNDU3XSL6OY3A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.433650237 (14h20m26.177707112s) [noise=true , address=104.36.82.43:4854 , piece_id=CY3WAFZSTIQ2AV6DYPIOZSOORUBPL3FBZK4CZNWL3SMOVG7DSZGQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.434316480 (14h20m26.178373354s) [noise=true , address=64.49.13.26:4948 , piece_id=TN7TAN4AYCIDUXZHESWSGMPXYDEM3ZN5RIS5HV3OVOALIFLJDQ7A, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.443916849 (14h20m26.187973724s) [noise=true , address=104.36.82.61:4393 , piece_id=MIKYUC5I7GASL3XCYWW7QHLYNJIKSLIHAPNF6NMV45ACHHJLTJJQ, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:04.444505957 (14h20m26.188562832s) [noise=true , address=45.91.116.6:4982 , piece_id=M3LYDOCIFKY5YHDBN6ZX6XGSUASZP74QRPZX3VFYSWD37QGHFNBA, seg_pos={0 0}, upload=14759]: Uploading piece... Aug 8 00:00:05.200540577 (14h20m26.944597434s) [upload=14759]: Next returned nil segment Aug 8 00:00:05.200687222 (14h20m26.944744097s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:05.355754277 (14h20m27.099811131s) [noise=true , address=104.36.82.59:4318 , piece_id=EHUOWZNGTRHSK7HXYEYYREX66ES4MPYKEU34EW66PFUPHI7VS3XQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.358198826 (14h20m27.102255692s) [noise=true , address=104.36.82.58:4268 , piece_id=YFPLOYXHUU3SOZJHFEMSTCNBM55AYJ5NONP7GLTDFU3OTZTW4BIQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.386961372 (14h20m27.13101822s) [noise=true , address=104.36.82.19:4404 , piece_id=LHQKAL4W24GGRXIZJ672W6CXY3NDMUN6NJVH2SFRHETKJMEHW62Q, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.391547939 (14h20m27.135604807s) [noise=true , address=104.36.82.55:4221 , piece_id=G3QN6J3Q4ZSDPAESESJMAWK3ICJLFBWPWY76QGCSMTHKYP4SUJ6Q, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.391724549 (14h20m27.135781419s) [noise=true , address=104.36.82.43:4854 , piece_id=CY3WAFZSTIQ2AV6DYPIOZSOORUBPL3FBZK4CZNWL3SMOVG7DSZGQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.625387364 (14h20m27.369444214s) [noise=true , address=104.36.82.17:4335 , piece_id=TB54EDHJ5XMYQPXWQB2NMUECRW5VTECH7H6KOPLCFUJ5SXS7U4OQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.685379596 (14h20m27.429436466s) [noise=true , address=104.36.82.38:4713 , piece_id=44IRUAVNYKZVR74WG64GQPGH3J337QYLGYWC3BUOZ3VS5NPJL27A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.696642486 (14h20m27.440699351s) [noise=true , address=45.91.116.6:4982 , piece_id=M3LYDOCIFKY5YHDBN6ZX6XGSUASZP74QRPZX3VFYSWD37QGHFNBA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.722664674 (14h20m27.466721544s) [noise=true , address=104.36.82.3:4796 , piece_id=APJB3DPBNQBOFK34LZ5MPOOR45ZXRQSMFWB4SOCHFOJCSDW3GWVQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.733358308 (14h20m27.477415158s) [noise=true , address=104.36.82.62:4425 , piece_id=BOX4GUARMRXU5IIE64CELJHQZFT6UWG4TFDATZHX5YFCJTFV2N4A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.781882083 (14h20m27.525938951s) [noise=true , address=45.91.116.14:4195 , piece_id=OFPVRGPPQRMNV3FV46GHPFWSXSNSMDDM7MZ7MFO5D4SP7NS6UZGA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:05.908897662 (14h20m27.652954524s) [noise=true , address=79.127.201.214:4295 , piece_id=CCDJBXRBUJ3NL4PDRRCTXBYXWYBIPYSNU3T3I6R4CTJWJRZTGGQA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:06.052322455 (14h20m27.796379308s) [noise=true , address=64.49.13.139:4470 , piece_id=WOXVGGQF2Q5PM6V3LGUOMMA4ZU7UPVW2KOR25TQ32PXN6OWCGGLA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:06.059613546 (14h20m27.803670401s) [noise=true , address=64.49.13.155:4916 , piece_id=54U5CI5CPB7DR56LS2GCOWNANROABC65PXTMSSVUTMZDQEZJY7CA, seg_pos={0 0}, upload=14759]: Done uploading piece. err: <nil> Aug 8 00:00:06.059650800 (14h20m27.803707652s) [seg_pos={0 0}, upload=14759]: Segment reached optimal threshold of 14 pieces. Aug 8 00:00:06.061105436 (14h20m27.805162307s) [noise=true , address=64.49.13.138:4490 , piece_id=VWBCADPZQBKS6HWUJZ6YWK7FDXN6Z2TSBJKK4Z7P3SDYS476UG5A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 8 00:00:06.061315764 (14h20m27.805372629s) [noise=true , address=64.49.13.26:4948 , piece_id=TN7TAN4AYCIDUXZHESWSGMPXYDEM3ZN5RIS5HV3OVOALIFLJDQ7A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVV4rAAZRd4DcLsakTTyPRNWck7AjBA6StKvsVLH2dCKVkwHB): protocol: expected piece hash; context canceled; EOF Aug 8 00:00:06.061491394 (14h20m27.805548286s) [noise=true , address=104.36.82.44:4885 , piece_id=ZHPCTPQUPDXJDPZQPBFSOD6KYKAVCHXQNXESSKEXNDU3XSL6OY3A, seg_pos={0 0}, upload=14759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: write tcp 139.59.144.22:35754->104.36.82.44:4885: use of closed network connection; write tcp 139.59.144.22:35754->104.36.82.44:4885: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35754->104.36.82.44:4885: use of closed network connection Aug 8 00:00:06.078661046 (14h20m27.822717918s) [noise=true , address=104.36.82.61:4393 , piece_id=MIKYUC5I7GASL3XCYWW7QHLYNJIKSLIHAPNF6NMV45ACHHJLTJJQ, seg_pos={0 0}, upload=14759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xfCSdYjRRxMaTw47dQmaXzGmMWyV1oCCGx7JdhvGqaaoN14Ff): protocol: write tcp 139.59.144.22:48102->104.36.82.61:4393: use of closed network connection; write tcp 139.59.144.22:48102->104.36.82.61:4393: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48102->104.36.82.61:4393: use of closed network connection Aug 8 00:00:06.078739080 (14h20m27.822795929s) [seg_pos={0 0}, upload=14759]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:00:06.078890768 (14h20m27.82294764s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:00:06.078899977 (14h20m27.82295685s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:06.568180745 (14h20m28.312237613s) [upload=14759]: Done uploading segments... Aug 8 00:00:06.568206007 (14h20m28.312262863s) [upload=14759]: Upload finished. err: <nil> Duration:* 2.85123802s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 291.192548ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-08 00:01:35 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 291.252875ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.XX.X.........................................X.XXX.X.X.....] duration: 932.542329ms FAIL (2025-08-08 00:01:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 932.574338ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 14 out of 60 is failed (threshold=3) [XXXXX..XX.X...................................X...XX.X...XX.] duration: 860.060112ms FAIL (2025-08-08 00:01:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 140355 (14h13m13.036197215s) [upload=14641]: Starting upload Aug 7 23:52:51.292216133 (14h13m13.036272985s) [upload=14641]: Done starting upload Aug 7 23:52:51.292307446 (14h13m13.036364307s) [upload=14641]: Uploading segments... Aug 7 23:52:51.292389087 (14h13m13.036445956s) [upload=14641]: Got next segment. Inline: true Aug 7 23:52:51.292394933 (14h13m13.036451793s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:52:51.292456032 (14h13m13.036512883s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:51.292461787 (14h13m13.036518647s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:51.292467676 (14h13m13.036524537s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:51.436915828 (14h13m13.180972678s) [upload=14641]: Done uploading segments... Aug 7 23:52:51.436934458 (14h13m13.18099131s) [upload=14641]: Upload finished. err: <nil> Aug 7 23:53:10.946261153 (14h13m32.690318028s) [upload=14646]: Starting upload Aug 7 23:53:10.946350198 (14h13m32.690407065s) [upload=14646]: Done starting upload Aug 7 23:53:10.946435423 (14h13m32.690492276s) [upload=14646]: Uploading segments... Aug 7 23:53:10.946489255 (14h13m32.690546124s) [upload=14646]: Got next segment. Inline: true Aug 7 23:53:10.946497660 (14h13m32.690554519s) [upload=14646]: Waiting for error group managing segments... Aug 7 23:53:10.946542678 (14h13m32.690599529s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:10.946548267 (14h13m32.690605128s) [upload=14646]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:10.946553792 (14h13m32.690610652s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:14.554651436 (14h13m36.298708309s) [upload=14646]: Done uploading segments... Aug 7 23:53:14.554675183 (14h13m36.298732045s) [upload=14646]: Upload finished. err: <nil> Aug 7 23:53:53.584502029 (14h14m15.328558897s) [upload=14658]: Starting upload Aug 7 23:53:53.584599757 (14h14m15.328656616s) [upload=14658]: Done starting upload Aug 7 23:53:53.584702364 (14h14m15.328759218s) [upload=14658]: Uploading segments... Aug 7 23:53:53.584798610 (14h14m15.32885548s) [upload=14658]: Got next segment. Inline: true Aug 7 23:53:53.584806594 (14h14m15.328863461s) [upload=14658]: Waiting for error group managing segments... Aug 7 23:53:53.584875193 (14h14m15.328932062s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:53.584882324 (14h14m15.328939193s) [upload=14658]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:53.584901657 (14h14m15.328958528s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:23.318821682 (14h14m45.062878542s) [upload=14658]: Done uploading segments... Aug 7 23:54:23.318863033 (14h14m45.062919884s) [upload=14658]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:36.775209359 (14h14m58.519266209s) [upload=14671]: Starting upload Aug 7 23:54:36.775292235 (14h14m58.519349112s) [upload=14671]: Done starting upload Aug 7 23:54:36.775412031 (14h14m58.519468883s) [upload=14671]: Uploading segments... Aug 7 23:54:36.775480767 (14h14m58.519537636s) [upload=14671]: Got next segment. Inline: true Aug 7 23:54:36.775487320 (14h14m58.519544181s) [upload=14671]: Waiting for error group managing segments... Aug 7 23:54:36.775539752 (14h14m58.519596612s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:36.775545893 (14h14m58.519602753s) [upload=14671]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:36.775549619 (14h14m58.519606479s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:36.926276427 (14h14m58.670333277s) [upload=14671]: Done uploading segments... Aug 7 23:54:36.926295206 (14h14m58.670352057s) [upload=14671]: Upload finished. err: <nil> Aug 7 23:54:57.675347843 (14h15m19.419404694s) [upload=14676]: Starting upload Aug 7 23:54:57.675412352 (14h15m19.419469212s) [upload=14676]: Done starting upload Aug 7 23:54:57.675475670 (14h15m19.41953252s) [upload=14676]: Uploading segments... Aug 7 23:54:57.675516336 (14h15m19.419573205s) [upload=14676]: Got next segment. Inline: true Aug 7 23:54:57.675521071 (14h15m19.419577931s) [upload=14676]: Waiting for error group managing segments... Aug 7 23:54:57.675565512 (14h15m19.419622375s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:57.675574255 (14h15m19.419631202s) [upload=14676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:57.675580694 (14h15m19.419637563s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:27.407552485 (14h15m49.151609344s) [upload=14676]: Done uploading segments... Aug 7 23:55:27.407592978 (14h15m49.151649829s) [upload=14676]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:06.851646232 (14h16m28.595703082s) [upload=14696]: Starting upload Aug 7 23:56:06.851717330 (14h16m28.595774191s) [upload=14696]: Done starting upload Aug 7 23:56:06.851786008 (14h16m28.59584286s) [upload=14696]: Uploading segments... Aug 7 23:56:06.851834232 (14h16m28.595891101s) [upload=14696]: Got next segment. Inline: true Aug 7 23:56:06.851839675 (14h16m28.595896535s) [upload=14696]: Waiting for error group managing segments... Aug 7 23:56:06.851896275 (14h16m28.595953125s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:06.851901659 (14h16m28.59595852s) [upload=14696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:06.851905505 (14h16m28.595962364s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:07.006202008 (14h16m28.750258868s) [upload=14696]: Done uploading segments... Aug 7 23:56:07.006222000 (14h16m28.750278861s) [upload=14696]: Upload finished. err: <nil> Aug 7 23:56:59.020763574 (14h17m20.764820421s) [upload=14712]: Starting upload Aug 7 23:56:59.020858955 (14h17m20.764915806s) [upload=14712]: Done starting upload Aug 7 23:56:59.020935444 (14h17m20.764992295s) [upload=14712]: Uploading segments... Aug 7 23:56:59.021007501 (14h17m20.76506437s) [upload=14712]: Got next segment. Inline: true Aug 7 23:56:59.021013631 (14h17m20.765070491s) [upload=14712]: Waiting for error group managing segments... Aug 7 23:56:59.021070471 (14h17m20.76512733s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:59.021076966 (14h17m20.765133821s) [upload=14712]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:59.021081305 (14h17m20.76513816s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:59.154820708 (14h17m20.898877569s) [upload=14712]: Done uploading segments... Aug 7 23:56:59.154839835 (14h17m20.898896685s) [upload=14712]: Upload finished. err: <nil> Aug 7 23:58:10.963677657 (14h18m32.707734508s) [upload=14728]: Starting upload Aug 7 23:58:10.963770332 (14h18m32.70782718s) [upload=14728]: Done starting upload Aug 7 23:58:10.963851725 (14h18m32.707908586s) [upload=14728]: Uploading segments... Aug 7 23:58:10.963901559 (14h18m32.707958431s) [upload=14728]: Got next segment. Inline: true Aug 7 23:58:10.963907663 (14h18m32.707964525s) [upload=14728]: Waiting for error group managing segments... Aug 7 23:58:10.963960167 (14h18m32.708017032s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:10.963966034 (14h18m32.708022896s) [upload=14728]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:10.963969944 (14h18m32.708026806s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.106069715 (14h18m32.850126571s) [upload=14728]: Done uploading segments... Aug 7 23:58:11.106112099 (14h18m32.850168961s) [upload=14728]: Upload finished. err: <nil> Aug 7 23:59:19.078128070 (14h19m40.822184931s) [upload=14749]: Starting upload Aug 7 23:59:19.078217499 (14h19m40.822274359s) [upload=14749]: Done starting upload Aug 7 23:59:19.078291040 (14h19m40.822347895s) [upload=14749]: Uploading segments... Aug 7 23:59:19.078333863 (14h19m40.822390732s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:19.078339134 (14h19m40.822395984s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:19.078391868 (14h19m40.822448717s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:19.078397213 (14h19m40.822454064s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:19.078401054 (14h19m40.822457905s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:48.808547234 (14h20m10.552604081s) [upload=14749]: Done uploading segments... Aug 7 23:59:48.808568563 (14h20m10.552625409s) [upload=14749]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:20.896255037 (14h20m42.640311905s) [upload=14765]: Starting upload Aug 8 00:00:20.896381648 (14h20m42.640438517s) [upload=14765]: Done starting upload Aug 8 00:00:20.896532416 (14h20m42.640589282s) [upload=14765]: Uploading segments... Aug 8 00:00:20.896610121 (14h20m42.640666994s) [upload=14765]: Got next segment. Inline: true Aug 8 00:00:20.896618785 (14h20m42.640675649s) [upload=14765]: Waiting for error group managing segments... Aug 8 00:00:20.896682486 (14h20m42.640739601s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:20.896690306 (14h20m42.640747187s) [upload=14765]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:20.896694946 (14h20m42.640751814s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:50.607106781 (14h21m12.351163663s) [upload=14765]: Done uploading segments... Aug 8 00:00:50.607134684 (14h21m12.351191558s) [upload=14765]: Upload finished. err: metaclient: context deadline exceeded Duration: 860.094139ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.806417905s FAIL (2025-08-08 00:01:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.806438697s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 120.12443ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-08 00:01:21 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 120.217139ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 868.472885ms FAIL (2025-08-08 00:01:17 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14121]: Done uploading segments... Aug 7 23:22:28.505845197 (13h42m50.249902044s) [upload=14121]: Upload finished. err: <nil> Aug 7 23:23:10.377483605 (13h43m32.121540458s) [upload=14132]: Starting upload Aug 7 23:23:10.377535635 (13h43m32.121592495s) [upload=14132]: Done starting upload Aug 7 23:23:10.377603797 (13h43m32.121660647s) [upload=14132]: Uploading segments... Aug 7 23:23:10.377638935 (13h43m32.121695804s) [upload=14132]: Got next segment. Inline: true Aug 7 23:23:10.377643594 (13h43m32.121700454s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:23:10.377688252 (13h43m32.121745104s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:23:10.377693427 (13h43m32.121750286s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:23:10.377696805 (13h43m32.121753666s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:23:10.562260724 (13h43m32.306317573s) [upload=14132]: Done uploading segments... Aug 7 23:23:10.562279228 (13h43m32.306336078s) [upload=14132]: Upload finished. err: <nil> Aug 7 23:24:07.896404858 (13h44m29.640461721s) [upload=14147]: Starting upload Aug 7 23:24:07.896490216 (13h44m29.640547079s) [upload=14147]: Done starting upload Aug 7 23:24:07.896580488 (13h44m29.640637339s) [upload=14147]: Uploading segments... Aug 7 23:24:07.896655006 (13h44m29.640711874s) [upload=14147]: Got next segment. Inline: true Aug 7 23:24:07.896660179 (13h44m29.640717039s) [upload=14147]: Waiting for error group managing segments... Aug 7 23:24:07.896714840 (13h44m29.6407717s) [upload=14147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:24:07.896719399 (13h44m29.64077626s) [upload=14147]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:24:07.896722726 (13h44m29.640779586s) [upload=14147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:24:08.048898130 (13h44m29.792954991s) [upload=14147]: Done uploading segments... Aug 7 23:24:08.048952526 (13h44m29.793009386s) [upload=14147]: Upload finished. err: <nil> Aug 7 23:24:53.794754379 (13h45m15.53881125s) [upload=14162]: Starting upload Aug 7 23:24:53.794842317 (13h45m15.538899177s) [upload=14162]: Done starting upload Aug 7 23:24:53.794935964 (13h45m15.538992814s) [upload=14162]: Uploading segments... Aug 7 23:24:53.795003752 (13h45m15.539060623s) [upload=14162]: Got next segment. Inline: true Aug 7 23:24:53.795010814 (13h45m15.539067676s) [upload=14162]: Waiting for error group managing segments... Aug 7 23:24:53.795066607 (13h45m15.539123488s) [upload=14162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:24:53.795158449 (13h45m15.5392153s) [upload=14162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:24:53.795163166 (13h45m15.539220026s) [upload=14162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:24:53.946994982 (13h45m15.691051833s) [upload=14162]: Done uploading segments... Aug 7 23:24:53.947013862 (13h45m15.691070711s) [upload=14162]: Upload finished. err: <nil> Aug 7 23:25:40.029856966 (13h46m1.773913836s) [upload=14177]: Starting upload Aug 7 23:25:40.029986938 (13h46m1.774043809s) [upload=14177]: Done starting upload Aug 7 23:25:40.030076796 (13h46m1.774133642s) [upload=14177]: Uploading segments... Aug 7 23:25:40.030132567 (13h46m1.774189438s) [upload=14177]: Got next segment. Inline: true Aug 7 23:25:40.030139885 (13h46m1.774196757s) [upload=14177]: Waiting for error group managing segments... Aug 7 23:25:40.030197588 (13h46m1.774254461s) [upload=14177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:25:40.030204146 (13h46m1.774261016s) [upload=14177]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:25:40.030207984 (13h46m1.77426485s) [upload=14177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:25:40.184198742 (13h46m1.928255592s) [upload=14177]: Done uploading segments... Aug 7 23:25:40.184219017 (13h46m1.928275867s) [upload=14177]: Upload finished. err: <nil> Aug 7 23:26:54.625649110 (13h47m16.369705971s) [upload=14198]: Starting upload Aug 7 23:26:54.625738563 (13h47m16.369795424s) [upload=14198]: Done starting upload Aug 7 23:26:54.625842691 (13h47m16.369899539s) [upload=14198]: Uploading segments... Aug 7 23:26:54.626016839 (13h47m16.37007371s) [upload=14198]: Got next segment. Inline: true Aug 7 23:26:54.626052112 (13h47m16.370108973s) [upload=14198]: Waiting for error group managing segments... Aug 7 23:26:54.626129780 (13h47m16.370186645s) [upload=14198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:26:54.626137392 (13h47m16.370194264s) [upload=14198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:26:54.626143182 (13h47m16.370200048s) [upload=14198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:26:54.775804171 (13h47m16.519861021s) [upload=14198]: Done uploading segments... Aug 7 23:26:54.775824249 (13h47m16.519881102s) [upload=14198]: Upload finished. err: <nil> Aug 7 23:27:32.263746490 (13h47m54.007803361s) [upload=14208]: Starting upload Aug 7 23:27:32.263833655 (13h47m54.007890501s) [upload=14208]: Done starting upload Aug 7 23:27:32.263976536 (13h47m54.008033385s) [upload=14208]: Uploading segments... Aug 7 23:27:32.264070926 (13h47m54.008127796s) [upload=14208]: Got next segment. Inline: true Aug 7 23:27:32.264080356 (13h47m54.00813733s) [upload=14208]: Waiting for error group managing segments... Aug 7 23:27:32.264154657 (13h47m54.008211526s) [upload=14208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:27:32.264163587 (13h47m54.008220454s) [upload=14208]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:27:32.264172017 (13h47m54.008228914s) [upload=14208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:27:32.490188559 (13h47m54.23424541s) [upload=14208]: Done uploading segments... Aug 7 23:27:32.490207326 (13h47m54.234264172s) [upload=14208]: Upload finished. err: <nil> Aug 7 23:27:57.927940477 (13h48m19.671997323s) [upload=14217]: Starting upload Aug 7 23:27:57.928016423 (13h48m19.672073283s) [upload=14217]: Done starting upload Aug 7 23:27:57.928101290 (13h48m19.672158142s) [upload=14217]: Uploading segments... Aug 7 23:27:57.928155692 (13h48m19.672212566s) [upload=14217]: Got next segment. Inline: true Aug 7 23:27:57.928162009 (13h48m19.672218869s) [upload=14217]: Waiting for error group managing segments... Aug 7 23:27:57.928213294 (13h48m19.672270145s) [upload=14217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:27:57.928219212 (13h48m19.672276074s) [upload=14217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:27:57.928223521 (13h48m19.672280383s) [upload=14217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:27:58.087544005 (13h48m19.831600874s) [upload=14217]: Done uploading segments... Aug 7 23:27:58.087570226 (13h48m19.831627094s) [upload=14217]: Upload finished. err: <nil> Aug 7 23:29:12.765200935 (13h49m34.509257795s) [upload=14237]: Starting upload Aug 7 23:29:12.765253106 (13h49m34.509309977s) [upload=14237]: Done starting upload Aug 7 23:29:12.765324641 (13h49m34.509381499s) [upload=14237]: Uploading segments... Aug 7 23:29:12.765374187 (13h49m34.509431056s) [upload=14237]: Got next segment. Inline: true Aug 7 23:29:12.765378947 (13h49m34.509435807s) [upload=14237]: Waiting for error group managing segments... Aug 7 23:29:12.765430859 (13h49m34.50948772s) [upload=14237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:29:12.765438106 (13h49m34.50949497s) [upload=14237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:29:12.765443127 (13h49m34.509499988s) [upload=14237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:29:12.926437674 (13h49m34.670494535s) [upload=14237]: Done uploading segments... Aug 7 23:29:12.926457418 (13h49m34.670514268s) [upload=14237]: Upload finished. err: <nil> Aug 7 23:29:42.225760316 (13h50m3.969817177s) [upload=14244]: Starting upload Aug 7 23:29:42.225855063 (13h50m3.969911929s) [upload=14244]: Done starting upload Aug 7 23:29:42.225962489 (13h50m3.970019355s) [upload=14244]: Uploading segments... Aug 7 23:29:42.226078484 (13h50m3.970135331s) [upload=14244]: Got next segment. Inline: true Aug 7 23:29:42.226083996 (13h50m3.970140842s) [upload=14244]: Waiting for error group managing segments... Aug 7 23:29:42.226143067 (13h50m3.970199914s) [upload=14244]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:29:42.226148511 (13h50m3.970205371s) [upload=14244]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:29:42.226153006 (13h50m3.970209865s) [upload=14244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:29:42.401822947 (13h50m4.14587981s) [upload=14244]: Done uploading segments... Aug 7 23:29:42.401847556 (13h50m4.145904405s) [upload=14244]: Upload finished. err: <nil> Aug 7 23:30:31.529840301 (13h50m53.273897165s) [upload=14254]: Starting upload Aug 7 23:30:31.530002531 (13h50m53.274059378s) [upload=14254]: Done starting upload Aug 7 23:30:31.530201544 (13h50m53.274258391s) [upload=14254]: Uploading segments... Aug 7 23:30:31.530298120 (13h50m53.274354985s) [upload=14254]: Got next segment. Inline: true Aug 7 23:30:31.530303457 (13h50m53.274360317s) [upload=14254]: Waiting for error group managing segments... Aug 7 23:30:31.530353721 (13h50m53.274410567s) [upload=14254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:30:31.530358927 (13h50m53.274415788s) [upload=14254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:30:31.530363758 (13h50m53.274420619s) [upload=14254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:30:31.696350967 (13h50m53.440407828s) [upload=14254]: Done uploading segments... Aug 7 23:30:31.696369460 (13h50m53.440426307s) [upload=14254]: Upload finished. err: <nil> Duration: 868.657406ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [X.XX.X.........................................X.XXX.X.X....] duration: 921.281934ms FAIL (2025-08-08 00:01:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 921.318898ms core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 336.53943ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-08 00:01:14 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 336.565178ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXX.XX........................................X.XX...XX.X..] duration: 2.54245756s FAIL (2025-08-08 00:01:07 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.542493629s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)