Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..............X.....................] duration: 2.842426805s FAIL (2025-08-08 00:30:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== oad Aug 8 00:12:40.688890451 (14h33m2.432947302s) [upload=14976]: Uploading segments... Aug 8 00:12:40.689315108 (14h33m2.433371968s) [upload=14976]: Got next segment. Inline: false Aug 8 00:12:40.689365721 (14h33m2.433422571s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:12:40.689371637 (14h33m2.433428498s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:12:40.938712718 (14h33m2.682769583s) [seg_pos={0 0}, upload=14976]: Begin upload segment... Aug 8 00:12:40.938780540 (14h33m2.68283739s) [seg_pos={0 0}, upload=14976]: Done begin upload segment. Aug 8 00:12:40.938826511 (14h33m2.682883372s) [noise=true , address=104.36.82.11:4122 , piece_id=T4ATL4PZUGFLXWE47UGHDXX3ID2LEJKQRGQQMF2G4J42TET2JVYQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.938837427 (14h33m2.682894292s) [noise=true , address=64.49.13.169:4715 , piece_id=TKX4G5GWLDNDPR66UADVNNWV2TAFGCVEMWZAU4GROJXFX3Q537TA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939397428 (14h33m2.683454296s) [noise=true , address=104.36.82.44:4881 , piece_id=TS5LURERTPPJMH3KOK3ABNKEWSWOF7IUNNKP5DONPQTXKIF4S55Q, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939417193 (14h33m2.683474059s) [noise=true , address=104.36.82.5:4859 , piece_id=FHBE3H7NTIY3PHCICRNYDGY7MEILS27N4O5DLKQQWM5UXELX3EWQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939726153 (14h33m2.683783022s) [noise=true , address=104.36.82.45:4937 , piece_id=4Z6T6K7EP3LS3XYSTOARORNVPGKX6HYIHV33I665O5FI4OPGOXNA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939786076 (14h33m2.683842943s) [noise=true , address=104.36.82.23:4453 , piece_id=V7D5MY7E63SEALXMVDJOS26NYQDQAFJ4SHUISCK6TPKDQKKI4Z3Q, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940057147 (14h33m2.68411401s) [noise=true , address=104.36.82.22:4537 , piece_id=3Q4MNZBN5UPYVL4BC673XQVPMLW3MYTY2CNLKFE7NW6ZWSFHAWPQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940208252 (14h33m2.684265105s) [noise=true , address=104.36.82.59:4313 , piece_id=LIWITE4IJVNHLO2DFFDI2M4D67UAHLM223EF6FAXGNGWIIMGH7FA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940408469 (14h33m2.68446533s) [noise=true , address=64.49.13.154:4930 , piece_id=OGQJV2HMKOE6IPET6A6JRENBWMGJ6NWSGL7Q55653IB6QBAUARQA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940742041 (14h33m2.684798902s) [noise=true , address=45.91.116.7:4034 , piece_id=XJQV5V55F3KLAF5MZZMWZN7HE6TRHQ2V6BDDMKOWMDMKEBAP6UZQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941021709 (14h33m2.685078579s) [noise=true , address=104.36.82.37:4671 , piece_id=HYA6VNL24EERU27SJLYO7TCV7DQIIH7Q4LXAEA5BXSHBTIXCHKDQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941299121 (14h33m2.685355986s) [noise=true , address=104.36.82.43:4855 , piece_id=AMNGI7UV2PCEQ2U4LAOXIFT4T3WXPD7YHLR77VR7MSSQGHESY5EA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941588088 (14h33m2.685644951s) [noise=true , address=104.36.82.19:4401 , piece_id=NACQ6PBCDIHNJAQIFQUVYAPU2HY5C4ZDW6QMJBGY3E45TNWHFQNQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941637056 (14h33m2.685693925s) [noise=true , address=104.36.82.61:4404 , piece_id=F5YUVKGDN4MPHJKOLR5DRVABOU6326LGWAJXKL4RCWZI35TLMXKQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941950914 (14h33m2.686007788s) [noise=true , address=79.127.201.215:4351 , piece_id=DR37C77MS5YTW7DK7XOJQGMEXWOSUKOD76SZY7TXLIB5RLXLEVMQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.942345807 (14h33m2.686402678s) [noise=true , address=104.36.82.55:4206 , piece_id=FEFQ3F2EFHMZI53KUIAGHESBATXL5S2EOAARLOHJ2YDSMGVLZRQQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.942560802 (14h33m2.686617667s) [noise=true , address=104.36.82.12:4158 , piece_id=SCBLDCSKC6HTOUIUA43BA4452TPHEIVKCMOSVHR42YXVUWYUFPRA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.943202713 (14h33m2.687259576s) [noise=true , address=193.149.164.143:4693 , piece_id=P5YP7XDPKE2VYGTIJMAD6TV2LAPXJRN47E3PDPDDLVNI73VJGUBA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:41.525441834 (14h33m3.269498698s) [upload=14976]: Next returned nil segment Aug 8 00:12:41.525462664 (14h33m3.269519531s) [upload=14976]: Waiting for error group managing segments... Aug 8 00:12:41.699976519 (14h33m3.444033384s) [noise=true , address=104.36.82.19:4401 , piece_id=NACQ6PBCDIHNJAQIFQUVYAPU2HY5C4ZDW6QMJBGY3E45TNWHFQNQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.700944219 (14h33m3.445001137s) [noise=true , address=104.36.82.23:4453 , piece_id=V7D5MY7E63SEALXMVDJOS26NYQDQAFJ4SHUISCK6TPKDQKKI4Z3Q, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.702379298 (14h33m3.446436158s) [noise=true , address=104.36.82.37:4671 , piece_id=HYA6VNL24EERU27SJLYO7TCV7DQIIH7Q4LXAEA5BXSHBTIXCHKDQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.710467855 (14h33m3.454524716s) [noise=true , address=104.36.82.55:4206 , piece_id=FEFQ3F2EFHMZI53KUIAGHESBATXL5S2EOAARLOHJ2YDSMGVLZRQQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.716048399 (14h33m3.46010525s) [noise=true , address=104.36.82.22:4537 , piece_id=3Q4MNZBN5UPYVL4BC673XQVPMLW3MYTY2CNLKFE7NW6ZWSFHAWPQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.769601981 (14h33m3.513658832s) [noise=true , address=104.36.82.44:4881 , piece_id=TS5LURERTPPJMH3KOK3ABNKEWSWOF7IUNNKP5DONPQTXKIF4S55Q, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.774083597 (14h33m3.518140448s) [noise=true , address=104.36.82.5:4859 , piece_id=FHBE3H7NTIY3PHCICRNYDGY7MEILS27N4O5DLKQQWM5UXELX3EWQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.790244473 (14h33m3.534301324s) [noise=true , address=45.91.116.7:4034 , piece_id=XJQV5V55F3KLAF5MZZMWZN7HE6TRHQ2V6BDDMKOWMDMKEBAP6UZQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.790597987 (14h33m3.534654837s) [noise=true , address=104.36.82.61:4404 , piece_id=F5YUVKGDN4MPHJKOLR5DRVABOU6326LGWAJXKL4RCWZI35TLMXKQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.814650455 (14h33m3.558707307s) [noise=true , address=104.36.82.59:4313 , piece_id=LIWITE4IJVNHLO2DFFDI2M4D67UAHLM223EF6FAXGNGWIIMGH7FA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.916929838 (14h33m3.660986699s) [noise=true , address=104.36.82.12:4158 , piece_id=SCBLDCSKC6HTOUIUA43BA4452TPHEIVKCMOSVHR42YXVUWYUFPRA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.946542933 (14h33m3.690599791s) [noise=true , address=79.127.201.215:4351 , piece_id=DR37C77MS5YTW7DK7XOJQGMEXWOSUKOD76SZY7TXLIB5RLXLEVMQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.014532160 (14h33m3.75858901s) [noise=true , address=104.36.82.43:4855 , piece_id=AMNGI7UV2PCEQ2U4LAOXIFT4T3WXPD7YHLR77VR7MSSQGHESY5EA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.157107757 (14h33m3.901164607s) [noise=true , address=104.36.82.11:4122 , piece_id=T4ATL4PZUGFLXWE47UGHDXX3ID2LEJKQRGQQMF2G4J42TET2JVYQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.157146638 (14h33m3.901203502s) [seg_pos={0 0}, upload=14976]: Segment reached optimal threshold of 14 pieces. Aug 8 00:12:42.157376876 (14h33m3.901433737s) [noise=true , address=64.49.13.169:4715 , piece_id=TKX4G5GWLDNDPR66UADVNNWV2TAFGCVEMWZAU4GROJXFX3Q537TA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157574941 (14h33m3.901631791s) [noise=true , address=193.149.164.143:4693 , piece_id=P5YP7XDPKE2VYGTIJMAD6TV2LAPXJRN47E3PDPDDLVNI73VJGUBA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LHoCE9CnM32tbMNqAB7HpmArSVLTycv9o6kSSFNUJsoDeERkQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157631621 (14h33m3.901688486s) [noise=true , address=64.49.13.154:4930 , piece_id=OGQJV2HMKOE6IPET6A6JRENBWMGJ6NWSGL7Q55653IB6QBAUARQA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kNK8y5et7MiiWNDT6iaC8YBAk3PX6WX1XSuU4QMSFBp3KymBb): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157680271 (14h33m3.901737121s) [noise=true , address=104.36.82.45:4937 , piece_id=4Z6T6K7EP3LS3XYSTOARORNVPGKX6HYIHV33I665O5FI4OPGOXNA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection; write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection Aug 8 00:12:42.157698358 (14h33m3.901755219s) [seg_pos={0 0}, upload=14976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:12:42.157816415 (14h33m3.901873289s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:12:42.157839617 (14h33m3.901896479s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:12:42.510566001 (14h33m4.254622851s) [upload=14976]: Done uploading segments... Aug 8 00:12:42.510584893 (14h33m4.254641743s) [upload=14976]: Upload finished. err: <nil> Duration: 2.842481012s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [..........XX..XX.X.X........................................] duration: 907.765822ms FAIL (2025-08-08 00:30:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0]: Starting upload Aug 7 23:53:17.153036870 (14h13m38.89709373s) [upload=14650]: Done starting upload Aug 7 23:53:17.153154678 (14h13m38.897211544s) [upload=14650]: Uploading segments... Aug 7 23:53:17.153233089 (14h13m38.897289957s) [upload=14650]: Got next segment. Inline: true Aug 7 23:53:17.153240256 (14h13m38.897297124s) [upload=14650]: Waiting for error group managing segments... Aug 7 23:53:17.153318892 (14h13m38.897375757s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:17.153325816 (14h13m38.89738268s) [upload=14650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:17.153330453 (14h13m38.897387314s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:16.875757982 (14h14m38.619814845s) [upload=14650]: Done uploading segments... Aug 7 23:54:16.875781849 (14h14m38.619838701s) [upload=14650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:24.158448119 (14h14m45.90250498s) [upload=14667]: Starting upload Aug 7 23:54:24.158525946 (14h14m45.902582806s) [upload=14667]: Done starting upload Aug 7 23:54:24.158598972 (14h14m45.902655824s) [upload=14667]: Uploading segments... Aug 7 23:54:24.158654148 (14h14m45.902711017s) [upload=14667]: Got next segment. Inline: true Aug 7 23:54:24.158659995 (14h14m45.902716856s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:54:24.158714896 (14h14m45.902771747s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:24.158720357 (14h14m45.902777209s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:24.158724105 (14h14m45.902780965s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:23.888400190 (14h15m45.63245704s) [upload=14667]: Done uploading segments... Aug 7 23:55:23.888419087 (14h15m45.632475938s) [upload=14667]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:24.297146309 (14h15m46.041203174s) [upload=14687]: Starting upload Aug 7 23:55:24.297232144 (14h15m46.04128901s) [upload=14687]: Done starting upload Aug 7 23:55:24.297315772 (14h15m46.041372619s) [upload=14687]: Uploading segments... Aug 7 23:55:24.297396238 (14h15m46.041453151s) [upload=14687]: Got next segment. Inline: true Aug 7 23:55:24.297403729 (14h15m46.04146059s) [upload=14687]: Waiting for error group managing segments... Aug 7 23:55:24.297444116 (14h15m46.041500976s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:24.297450971 (14h15m46.041507831s) [upload=14687]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:24.297453883 (14h15m46.041510749s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:24.439126204 (14h15m46.183183055s) [upload=14687]: Done uploading segments... Aug 7 23:55:24.439141121 (14h15m46.183197972s) [upload=14687]: Upload finished. err: <nil> Aug 7 23:56:03.068174127 (14h16m24.812230977s) [upload=14695]: Starting upload Aug 7 23:56:03.068258755 (14h16m24.812315617s) [upload=14695]: Done starting upload Aug 7 23:56:03.068368961 (14h16m24.812425812s) [upload=14695]: Uploading segments... Aug 7 23:56:03.068437851 (14h16m24.812494719s) [upload=14695]: Got next segment. Inline: true Aug 7 23:56:03.068445058 (14h16m24.812501927s) [upload=14695]: Waiting for error group managing segments... Aug 7 23:56:03.068516175 (14h16m24.812573049s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:03.068523275 (14h16m24.812580137s) [upload=14695]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:03.068527513 (14h16m24.812584488s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:03.370378590 (14h16m25.114435452s) [upload=14695]: Done uploading segments... Aug 7 23:56:03.370401742 (14h16m25.114458656s) [upload=14695]: Upload finished. err: <nil> Aug 7 23:56:41.636734874 (14h17m3.380791742s) [upload=14707]: Starting upload Aug 7 23:56:41.636844337 (14h17m3.380901201s) [upload=14707]: Done starting upload Aug 7 23:56:41.636938361 (14h17m3.380995223s) [upload=14707]: Uploading segments... Aug 7 23:56:41.637004566 (14h17m3.381061436s) [upload=14707]: Got next segment. Inline: true Aug 7 23:56:41.637011720 (14h17m3.381068581s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:56:41.637073380 (14h17m3.381130242s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:41.637079684 (14h17m3.381136546s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:41.637083892 (14h17m3.381140753s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:41.369074470 (14h18m3.113131318s) [upload=14707]: Done uploading segments... Aug 7 23:57:41.369096313 (14h18m3.11315316s) [upload=14707]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:41.785857243 (14h18m3.529914104s) [upload=14726]: Starting upload Aug 7 23:57:41.785973295 (14h18m3.530030155s) [upload=14726]: Done starting upload Aug 7 23:57:41.786053418 (14h18m3.530110268s) [upload=14726]: Uploading segments... Aug 7 23:57:41.786115407 (14h18m3.530172273s) [upload=14726]: Got next segment. Inline: true Aug 7 23:57:41.786121268 (14h18m3.530178127s) [upload=14726]: Waiting for error group managing segments... Aug 7 23:57:41.786162041 (14h18m3.530218901s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:41.786167247 (14h18m3.530224108s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:41.786170583 (14h18m3.530227443s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.518215592 (14h19m3.262272452s) [upload=14726]: Done uploading segments... Aug 7 23:58:41.518237426 (14h19m3.262294275s) [upload=14726]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:49.080161447 (14h19m10.824218292s) [upload=14741]: Starting upload Aug 7 23:58:49.080236560 (14h19m10.824293407s) [upload=14741]: Done starting upload Aug 7 23:58:49.080313897 (14h19m10.824370744s) [upload=14741]: Uploading segments... Aug 7 23:58:49.080371825 (14h19m10.824428691s) [upload=14741]: Got next segment. Inline: true Aug 7 23:58:49.080376725 (14h19m10.824433586s) [upload=14741]: Waiting for error group managing segments... Aug 7 23:58:49.080418334 (14h19m10.824475197s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:49.080422560 (14h19m10.824479419s) [upload=14741]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:49.080438323 (14h19m10.824495186s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:49.220444289 (14h19m10.964501139s) [upload=14741]: Done uploading segments... Aug 7 23:58:49.220462660 (14h19m10.96451951s) [upload=14741]: Upload finished. err: <nil> Aug 7 23:59:26.271206280 (14h19m48.015263132s) [upload=14752]: Starting upload Aug 7 23:59:26.271300842 (14h19m48.015357706s) [upload=14752]: Done starting upload Aug 7 23:59:26.271409542 (14h19m48.015466389s) [upload=14752]: Uploading segments... Aug 7 23:59:26.271471969 (14h19m48.015528839s) [upload=14752]: Got next segment. Inline: true Aug 7 23:59:26.271478422 (14h19m48.015535286s) [upload=14752]: Waiting for error group managing segments... Aug 7 23:59:26.271524304 (14h19m48.015581168s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.271531803 (14h19m48.01558867s) [upload=14752]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:26.271535740 (14h19m48.015592604s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.006104454 (14h20m47.750161323s) [upload=14752]: Done uploading segments... Aug 8 00:00:26.006140966 (14h20m47.750197816s) [upload=14752]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:26.408809053 (14h20m48.152865927s) [upload=14766]: Starting upload Aug 8 00:00:26.408886561 (14h20m48.152943434s) [upload=14766]: Done starting upload Aug 8 00:00:26.408975679 (14h20m48.15303253s) [upload=14766]: Uploading segments... Aug 8 00:00:26.409028539 (14h20m48.153085476s) [upload=14766]: Got next segment. Inline: true Aug 8 00:00:26.409034850 (14h20m48.153091712s) [upload=14766]: Waiting for error group managing segments... Aug 8 00:00:26.409085190 (14h20m48.153142042s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:26.409090930 (14h20m48.153147793s) [upload=14766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:26.409094742 (14h20m48.153151605s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.549853950 (14h20m48.2939108s) [upload=14766]: Done uploading segments... Aug 8 00:00:26.549878419 (14h20m48.293935265s) [upload=14766]: Upload finished. err: <nil> Aug 8 00:01:08.274172938 (14h21m30.018229799s) [upload=14780]: Starting upload Aug 8 00:01:08.274248443 (14h21m30.018305302s) [upload=14780]: Done starting upload Aug 8 00:01:08.274326739 (14h21m30.018383585s) [upload=14780]: Uploading segments... Aug 8 00:01:08.274380689 (14h21m30.018437549s) [upload=14780]: Got next segment. Inline: true Aug 8 00:01:08.274385704 (14h21m30.018442565s) [upload=14780]: Waiting for error group managing segments... Aug 8 00:01:08.274437335 (14h21m30.018494182s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:08.274442821 (14h21m30.018499667s) [upload=14780]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:08.274445609 (14h21m30.018502454s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:08.005343335 (14h22m29.749400197s) [upload=14780]: Done uploading segments... Aug 8 00:02:08.005365121 (14h22m29.749421982s) [upload=14780]: Upload finished. err: metaclient: context deadline exceeded Duration: 907.801627ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................X...X....................................] duration: 1.167217738s FAIL (2025-08-08 00:30:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.167254044s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............XXX..XX........................................] duration: 2.843689107s FAIL (2025-08-08 00:30:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 807513521 (14h17m32.551570367s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:55.062716703 (14h18m16.806773554s) [seg_pos={0 0}, upload=14717]: Begin upload segment... Aug 7 23:57:55.062772247 (14h18m16.806829106s) [seg_pos={0 0}, upload=14717]: Done begin upload segment. Aug 7 23:57:55.062817614 (14h18m16.806874477s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.062833275 (14h18m16.806890146s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063321348 (14h18m16.807378211s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063625241 (14h18m16.8076821s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.064591461 (14h18m16.808648334s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065249737 (14h18m16.809306608s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065844042 (14h18m16.809900909s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.066738904 (14h18m16.810795771s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067295868 (14h18m16.81135274s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067868039 (14h18m16.811924903s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068378475 (14h18m16.812435346s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068951556 (14h18m16.81300842s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069355853 (14h18m16.813412717s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069707384 (14h18m16.813764245s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070103904 (14h18m16.814160763s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070434322 (14h18m16.814491182s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070775335 (14h18m16.814832199s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.072531247 (14h18m16.816588107s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.642344995 (14h18m17.386401842s) [upload=14717]: Next returned nil segment Aug 7 23:57:55.642365727 (14h18m17.38642259s) [upload=14717]: Waiting for error group managing segments... Aug 7 23:57:55.820297514 (14h18m17.564354366s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.832273438 (14h18m17.576330291s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.845342869 (14h18m17.58939972s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.850001106 (14h18m17.594057955s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874147776 (14h18m17.618204625s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874555951 (14h18m17.6186128s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.890084811 (14h18m17.634141672s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.901998021 (14h18m17.646054868s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.930296357 (14h18m17.674353212s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.008745690 (14h18m17.752802541s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.041830348 (14h18m17.785887195s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.086405192 (14h18m17.830462041s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.092724415 (14h18m17.83678127s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095750120 (14h18m17.839806985s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095784368 (14h18m17.839841231s) [seg_pos={0 0}, upload=14717]: Segment reached optimal threshold of 14 pieces. Aug 7 23:57:56.095960472 (14h18m17.840017334s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096160491 (14h18m17.840217355s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096190692 (14h18m17.840247554s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eu7SDhEXZGS6HeAXFZWfxoci2oPdjtYvZ5Bbhvae2neKTxYKo): protocol: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection Aug 7 23:57:56.096286416 (14h18m17.840343264s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w4v4QV7PxhXTwuxbvTQST2buoWNySrziRYTo9Mb8dpENd9XCX): protocol: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection Aug 7 23:57:56.096300170 (14h18m17.840357019s) [seg_pos={0 0}, upload=14717]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:57:56.096419305 (14h18m17.840476164s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:56.096426206 (14h18m17.84048307s) [upload=14717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:30.094636522 (14h22m51.838693376s) [upload=14717]: Done uploading segments... Aug 8 00:02:30.094667105 (14h22m51.838723955s) [upload=14717]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.843733641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........XX.X.X...XX.......................................] duration: 2.512917303s FAIL (2025-08-08 00:30:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 6]: Uploading piece... Aug 7 23:57:10.893816491 (14h17m32.637873354s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.894285524 (14h17m32.638342391s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895348932 (14h17m32.639405787s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895932061 (14h17m32.639988921s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.896545320 (14h17m32.640602184s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897117472 (14h17m32.641174341s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897621325 (14h17m32.641678192s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898101338 (14h17m32.642158204s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898693607 (14h17m32.642750474s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.900512475 (14h17m32.644569335s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.902337804 (14h17m32.646394669s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.904215667 (14h17m32.648272537s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.906034481 (14h17m32.650091365s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.907857426 (14h17m32.651914295s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.908666188 (14h17m32.652723069s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:11.510238622 (14h17m33.254295484s) [upload=14716]: Next returned nil segment Aug 7 23:57:11.510261464 (14h17m33.254318324s) [upload=14716]: Waiting for error group managing segments... Aug 7 23:57:11.657094388 (14h17m33.401151236s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.674408374 (14h17m33.418465234s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.681203586 (14h17m33.425260449s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.682780850 (14h17m33.4268377s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.684042315 (14h17m33.428099176s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.719658547 (14h17m33.463715401s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.730348391 (14h17m33.474405242s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.748264399 (14h17m33.492321249s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.769764028 (14h17m33.513820878s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.774177874 (14h17m33.518234733s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.789864472 (14h17m33.533921326s) [noise=true , address=45.91.116.13:4153 , piece_id=WBFFQT5I6ZR5ZKUJRFULNOJZ455KIJMA5EEATDDOCVFUKRD67NFA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.799519597 (14h17m33.543576447s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800753821 (14h17m33.544810667s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800785983 (14h17m33.544842842s) [seg_pos={0 0}, upload=14716]: Segment reached optimal threshold of 13 pieces. Aug 7 23:57:11.801026052 (14h17m33.545082898s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801098020 (14h17m33.545154867s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801174944 (14h17m33.54523179s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection Aug 7 23:57:11.801184926 (14h17m33.545241773s) [seg_pos={0 0}, upload=14716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:57:11.801257096 (14h17m33.545313956s) [upload=14716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:11.801261009 (14h17m33.545317869s) [upload=14716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:11.942784613 (14h17m33.686841513s) [upload=14716]: Done uploading segments... Aug 7 23:57:11.942815424 (14h17m33.686872279s) [upload=14716]: Upload finished. err: <nil> Aug 7 23:58:20.561350310 (14h18m42.305407184s) [upload=14732]: Starting upload Aug 7 23:58:20.561483402 (14h18m42.30554027s) [upload=14732]: Done starting upload Aug 7 23:58:20.561650297 (14h18m42.305707159s) [upload=14732]: Uploading segments... Aug 7 23:58:20.562196341 (14h18m42.306253212s) [upload=14732]: Got next segment. Inline: false Aug 7 23:58:20.562334166 (14h18m42.306391031s) [upload=14732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:20.562343255 (14h18m42.306400125s) [upload=14732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:50.281195862 (14h19m12.025252712s) [upload=14732]: Next segment err: context deadline exceeded Aug 7 23:58:50.281383705 (14h19m12.025440555s) [upload=14732]: Done uploading segments... Aug 7 23:58:50.281405600 (14h19m12.02546245s) [upload=14732]: Upload finished. err: context deadline exceeded Aug 7 23:58:50.716777496 (14h19m12.460834346s) [upload=14742]: Starting upload Aug 7 23:58:50.716857905 (14h19m12.460914751s) [upload=14742]: Done starting upload Aug 7 23:58:50.716999972 (14h19m12.461056825s) [upload=14742]: Uploading segments... Aug 7 23:58:50.717186124 (14h19m12.461242976s) [upload=14742]: Got next segment. Inline: false Aug 7 23:58:50.717245601 (14h19m12.461302466s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:50.717252185 (14h19m12.461309053s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:20.436529435 (14h19m42.180586305s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:20.436734882 (14h19m42.180791733s) [upload=14742]: Done uploading segments... Aug 7 23:59:20.436756200 (14h19m42.180813054s) [upload=14742]: Upload finished. err: context deadline exceeded Duration: 2.512975819s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 111.335675ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-08 00:30:01 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 111.367577ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X.XX...XX.X..X...................................] duration: 2.41062637s FAIL (2025-08-08 00:30:00 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.410662473s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X..............X....................] duration: 2.811482736s FAIL (2025-08-08 00:29:54 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== oad Aug 8 00:12:40.688890451 (14h33m2.432947302s) [upload=14976]: Uploading segments... Aug 8 00:12:40.689315108 (14h33m2.433371968s) [upload=14976]: Got next segment. Inline: false Aug 8 00:12:40.689365721 (14h33m2.433422571s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:12:40.689371637 (14h33m2.433428498s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:12:40.938712718 (14h33m2.682769583s) [seg_pos={0 0}, upload=14976]: Begin upload segment... Aug 8 00:12:40.938780540 (14h33m2.68283739s) [seg_pos={0 0}, upload=14976]: Done begin upload segment. Aug 8 00:12:40.938826511 (14h33m2.682883372s) [noise=true , address=104.36.82.11:4122 , piece_id=T4ATL4PZUGFLXWE47UGHDXX3ID2LEJKQRGQQMF2G4J42TET2JVYQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.938837427 (14h33m2.682894292s) [noise=true , address=64.49.13.169:4715 , piece_id=TKX4G5GWLDNDPR66UADVNNWV2TAFGCVEMWZAU4GROJXFX3Q537TA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939397428 (14h33m2.683454296s) [noise=true , address=104.36.82.44:4881 , piece_id=TS5LURERTPPJMH3KOK3ABNKEWSWOF7IUNNKP5DONPQTXKIF4S55Q, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939417193 (14h33m2.683474059s) [noise=true , address=104.36.82.5:4859 , piece_id=FHBE3H7NTIY3PHCICRNYDGY7MEILS27N4O5DLKQQWM5UXELX3EWQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939726153 (14h33m2.683783022s) [noise=true , address=104.36.82.45:4937 , piece_id=4Z6T6K7EP3LS3XYSTOARORNVPGKX6HYIHV33I665O5FI4OPGOXNA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939786076 (14h33m2.683842943s) [noise=true , address=104.36.82.23:4453 , piece_id=V7D5MY7E63SEALXMVDJOS26NYQDQAFJ4SHUISCK6TPKDQKKI4Z3Q, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940057147 (14h33m2.68411401s) [noise=true , address=104.36.82.22:4537 , piece_id=3Q4MNZBN5UPYVL4BC673XQVPMLW3MYTY2CNLKFE7NW6ZWSFHAWPQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940208252 (14h33m2.684265105s) [noise=true , address=104.36.82.59:4313 , piece_id=LIWITE4IJVNHLO2DFFDI2M4D67UAHLM223EF6FAXGNGWIIMGH7FA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940408469 (14h33m2.68446533s) [noise=true , address=64.49.13.154:4930 , piece_id=OGQJV2HMKOE6IPET6A6JRENBWMGJ6NWSGL7Q55653IB6QBAUARQA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940742041 (14h33m2.684798902s) [noise=true , address=45.91.116.7:4034 , piece_id=XJQV5V55F3KLAF5MZZMWZN7HE6TRHQ2V6BDDMKOWMDMKEBAP6UZQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941021709 (14h33m2.685078579s) [noise=true , address=104.36.82.37:4671 , piece_id=HYA6VNL24EERU27SJLYO7TCV7DQIIH7Q4LXAEA5BXSHBTIXCHKDQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941299121 (14h33m2.685355986s) [noise=true , address=104.36.82.43:4855 , piece_id=AMNGI7UV2PCEQ2U4LAOXIFT4T3WXPD7YHLR77VR7MSSQGHESY5EA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941588088 (14h33m2.685644951s) [noise=true , address=104.36.82.19:4401 , piece_id=NACQ6PBCDIHNJAQIFQUVYAPU2HY5C4ZDW6QMJBGY3E45TNWHFQNQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941637056 (14h33m2.685693925s) [noise=true , address=104.36.82.61:4404 , piece_id=F5YUVKGDN4MPHJKOLR5DRVABOU6326LGWAJXKL4RCWZI35TLMXKQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941950914 (14h33m2.686007788s) [noise=true , address=79.127.201.215:4351 , piece_id=DR37C77MS5YTW7DK7XOJQGMEXWOSUKOD76SZY7TXLIB5RLXLEVMQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.942345807 (14h33m2.686402678s) [noise=true , address=104.36.82.55:4206 , piece_id=FEFQ3F2EFHMZI53KUIAGHESBATXL5S2EOAARLOHJ2YDSMGVLZRQQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.942560802 (14h33m2.686617667s) [noise=true , address=104.36.82.12:4158 , piece_id=SCBLDCSKC6HTOUIUA43BA4452TPHEIVKCMOSVHR42YXVUWYUFPRA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.943202713 (14h33m2.687259576s) [noise=true , address=193.149.164.143:4693 , piece_id=P5YP7XDPKE2VYGTIJMAD6TV2LAPXJRN47E3PDPDDLVNI73VJGUBA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:41.525441834 (14h33m3.269498698s) [upload=14976]: Next returned nil segment Aug 8 00:12:41.525462664 (14h33m3.269519531s) [upload=14976]: Waiting for error group managing segments... Aug 8 00:12:41.699976519 (14h33m3.444033384s) [noise=true , address=104.36.82.19:4401 , piece_id=NACQ6PBCDIHNJAQIFQUVYAPU2HY5C4ZDW6QMJBGY3E45TNWHFQNQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.700944219 (14h33m3.445001137s) [noise=true , address=104.36.82.23:4453 , piece_id=V7D5MY7E63SEALXMVDJOS26NYQDQAFJ4SHUISCK6TPKDQKKI4Z3Q, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.702379298 (14h33m3.446436158s) [noise=true , address=104.36.82.37:4671 , piece_id=HYA6VNL24EERU27SJLYO7TCV7DQIIH7Q4LXAEA5BXSHBTIXCHKDQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.710467855 (14h33m3.454524716s) [noise=true , address=104.36.82.55:4206 , piece_id=FEFQ3F2EFHMZI53KUIAGHESBATXL5S2EOAARLOHJ2YDSMGVLZRQQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.716048399 (14h33m3.46010525s) [noise=true , address=104.36.82.22:4537 , piece_id=3Q4MNZBN5UPYVL4BC673XQVPMLW3MYTY2CNLKFE7NW6ZWSFHAWPQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.769601981 (14h33m3.513658832s) [noise=true , address=104.36.82.44:4881 , piece_id=TS5LURERTPPJMH3KOK3ABNKEWSWOF7IUNNKP5DONPQTXKIF4S55Q, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.774083597 (14h33m3.518140448s) [noise=true , address=104.36.82.5:4859 , piece_id=FHBE3H7NTIY3PHCICRNYDGY7MEILS27N4O5DLKQQWM5UXELX3EWQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.790244473 (14h33m3.534301324s) [noise=true , address=45.91.116.7:4034 , piece_id=XJQV5V55F3KLAF5MZZMWZN7HE6TRHQ2V6BDDMKOWMDMKEBAP6UZQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.790597987 (14h33m3.534654837s) [noise=true , address=104.36.82.61:4404 , piece_id=F5YUVKGDN4MPHJKOLR5DRVABOU6326LGWAJXKL4RCWZI35TLMXKQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.814650455 (14h33m3.558707307s) [noise=true , address=104.36.82.59:4313 , piece_id=LIWITE4IJVNHLO2DFFDI2M4D67UAHLM223EF6FAXGNGWIIMGH7FA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.916929838 (14h33m3.660986699s) [noise=true , address=104.36.82.12:4158 , piece_id=SCBLDCSKC6HTOUIUA43BA4452TPHEIVKCMOSVHR42YXVUWYUFPRA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.946542933 (14h33m3.690599791s) [noise=true , address=79.127.201.215:4351 , piece_id=DR37C77MS5YTW7DK7XOJQGMEXWOSUKOD76SZY7TXLIB5RLXLEVMQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.014532160 (14h33m3.75858901s) [noise=true , address=104.36.82.43:4855 , piece_id=AMNGI7UV2PCEQ2U4LAOXIFT4T3WXPD7YHLR77VR7MSSQGHESY5EA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.157107757 (14h33m3.901164607s) [noise=true , address=104.36.82.11:4122 , piece_id=T4ATL4PZUGFLXWE47UGHDXX3ID2LEJKQRGQQMF2G4J42TET2JVYQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.157146638 (14h33m3.901203502s) [seg_pos={0 0}, upload=14976]: Segment reached optimal threshold of 14 pieces. Aug 8 00:12:42.157376876 (14h33m3.901433737s) [noise=true , address=64.49.13.169:4715 , piece_id=TKX4G5GWLDNDPR66UADVNNWV2TAFGCVEMWZAU4GROJXFX3Q537TA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157574941 (14h33m3.901631791s) [noise=true , address=193.149.164.143:4693 , piece_id=P5YP7XDPKE2VYGTIJMAD6TV2LAPXJRN47E3PDPDDLVNI73VJGUBA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LHoCE9CnM32tbMNqAB7HpmArSVLTycv9o6kSSFNUJsoDeERkQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157631621 (14h33m3.901688486s) [noise=true , address=64.49.13.154:4930 , piece_id=OGQJV2HMKOE6IPET6A6JRENBWMGJ6NWSGL7Q55653IB6QBAUARQA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kNK8y5et7MiiWNDT6iaC8YBAk3PX6WX1XSuU4QMSFBp3KymBb): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157680271 (14h33m3.901737121s) [noise=true , address=104.36.82.45:4937 , piece_id=4Z6T6K7EP3LS3XYSTOARORNVPGKX6HYIHV33I665O5FI4OPGOXNA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection; write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection Aug 8 00:12:42.157698358 (14h33m3.901755219s) [seg_pos={0 0}, upload=14976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:12:42.157816415 (14h33m3.901873289s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:12:42.157839617 (14h33m3.901896479s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:12:42.510566001 (14h33m4.254622851s) [upload=14976]: Done uploading segments... Aug 8 00:12:42.510584893 (14h33m4.254641743s) [upload=14976]: Upload finished. err: <nil> Duration: 2.811518522s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............XXX..XX.......................................] duration: 2.904275814s FAIL (2025-08-08 00:29:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 807513521 (14h17m32.551570367s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:55.062716703 (14h18m16.806773554s) [seg_pos={0 0}, upload=14717]: Begin upload segment... Aug 7 23:57:55.062772247 (14h18m16.806829106s) [seg_pos={0 0}, upload=14717]: Done begin upload segment. Aug 7 23:57:55.062817614 (14h18m16.806874477s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.062833275 (14h18m16.806890146s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063321348 (14h18m16.807378211s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063625241 (14h18m16.8076821s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.064591461 (14h18m16.808648334s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065249737 (14h18m16.809306608s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065844042 (14h18m16.809900909s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.066738904 (14h18m16.810795771s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067295868 (14h18m16.81135274s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067868039 (14h18m16.811924903s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068378475 (14h18m16.812435346s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068951556 (14h18m16.81300842s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069355853 (14h18m16.813412717s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069707384 (14h18m16.813764245s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070103904 (14h18m16.814160763s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070434322 (14h18m16.814491182s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070775335 (14h18m16.814832199s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.072531247 (14h18m16.816588107s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.642344995 (14h18m17.386401842s) [upload=14717]: Next returned nil segment Aug 7 23:57:55.642365727 (14h18m17.38642259s) [upload=14717]: Waiting for error group managing segments... Aug 7 23:57:55.820297514 (14h18m17.564354366s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.832273438 (14h18m17.576330291s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.845342869 (14h18m17.58939972s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.850001106 (14h18m17.594057955s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874147776 (14h18m17.618204625s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874555951 (14h18m17.6186128s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.890084811 (14h18m17.634141672s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.901998021 (14h18m17.646054868s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.930296357 (14h18m17.674353212s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.008745690 (14h18m17.752802541s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.041830348 (14h18m17.785887195s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.086405192 (14h18m17.830462041s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.092724415 (14h18m17.83678127s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095750120 (14h18m17.839806985s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095784368 (14h18m17.839841231s) [seg_pos={0 0}, upload=14717]: Segment reached optimal threshold of 14 pieces. Aug 7 23:57:56.095960472 (14h18m17.840017334s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096160491 (14h18m17.840217355s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096190692 (14h18m17.840247554s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eu7SDhEXZGS6HeAXFZWfxoci2oPdjtYvZ5Bbhvae2neKTxYKo): protocol: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection Aug 7 23:57:56.096286416 (14h18m17.840343264s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w4v4QV7PxhXTwuxbvTQST2buoWNySrziRYTo9Mb8dpENd9XCX): protocol: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection Aug 7 23:57:56.096300170 (14h18m17.840357019s) [seg_pos={0 0}, upload=14717]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:57:56.096419305 (14h18m17.840476164s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:56.096426206 (14h18m17.84048307s) [upload=14717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:30.094636522 (14h22m51.838693376s) [upload=14717]: Done uploading segments... Aug 8 00:02:30.094667105 (14h22m51.838723955s) [upload=14717]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.904330002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.....X.XXX.X.X.....X........................................] duration: 902.897129ms FAIL (2025-08-08 00:29:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 902.934313ms core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................X...X...................................] duration: 1.123572139s FAIL (2025-08-08 00:29:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.123680431s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.675613692s FAIL (2025-08-08 00:29:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.675657885s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 127.682936ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-08 00:29:37 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 127.710486ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............XXX.X.X........................................] duration: 2.850178155s FAIL (2025-08-08 00:29:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.850230379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............XXX..XX......................................] duration: 3.343467189s FAIL (2025-08-08 00:29:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 807513521 (14h17m32.551570367s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:55.062716703 (14h18m16.806773554s) [seg_pos={0 0}, upload=14717]: Begin upload segment... Aug 7 23:57:55.062772247 (14h18m16.806829106s) [seg_pos={0 0}, upload=14717]: Done begin upload segment. Aug 7 23:57:55.062817614 (14h18m16.806874477s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.062833275 (14h18m16.806890146s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063321348 (14h18m16.807378211s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063625241 (14h18m16.8076821s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.064591461 (14h18m16.808648334s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065249737 (14h18m16.809306608s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065844042 (14h18m16.809900909s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.066738904 (14h18m16.810795771s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067295868 (14h18m16.81135274s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067868039 (14h18m16.811924903s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068378475 (14h18m16.812435346s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068951556 (14h18m16.81300842s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069355853 (14h18m16.813412717s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069707384 (14h18m16.813764245s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070103904 (14h18m16.814160763s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070434322 (14h18m16.814491182s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070775335 (14h18m16.814832199s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.072531247 (14h18m16.816588107s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.642344995 (14h18m17.386401842s) [upload=14717]: Next returned nil segment Aug 7 23:57:55.642365727 (14h18m17.38642259s) [upload=14717]: Waiting for error group managing segments... Aug 7 23:57:55.820297514 (14h18m17.564354366s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.832273438 (14h18m17.576330291s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.845342869 (14h18m17.58939972s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.850001106 (14h18m17.594057955s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874147776 (14h18m17.618204625s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874555951 (14h18m17.6186128s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.890084811 (14h18m17.634141672s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.901998021 (14h18m17.646054868s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.930296357 (14h18m17.674353212s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.008745690 (14h18m17.752802541s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.041830348 (14h18m17.785887195s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.086405192 (14h18m17.830462041s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.092724415 (14h18m17.83678127s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095750120 (14h18m17.839806985s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095784368 (14h18m17.839841231s) [seg_pos={0 0}, upload=14717]: Segment reached optimal threshold of 14 pieces. Aug 7 23:57:56.095960472 (14h18m17.840017334s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096160491 (14h18m17.840217355s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096190692 (14h18m17.840247554s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eu7SDhEXZGS6HeAXFZWfxoci2oPdjtYvZ5Bbhvae2neKTxYKo): protocol: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection Aug 7 23:57:56.096286416 (14h18m17.840343264s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w4v4QV7PxhXTwuxbvTQST2buoWNySrziRYTo9Mb8dpENd9XCX): protocol: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection Aug 7 23:57:56.096300170 (14h18m17.840357019s) [seg_pos={0 0}, upload=14717]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:57:56.096419305 (14h18m17.840476164s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:56.096426206 (14h18m17.84048307s) [upload=14717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:30.094636522 (14h22m51.838693376s) [upload=14717]: Done uploading segments... Aug 8 00:02:30.094667105 (14h22m51.838723955s) [upload=14717]: Upload finished. err: metaclient: context deadline exceeded Duration:* 3.343505183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.........X...XX.X...XX......................................] duration: 879.677486ms FAIL (2025-08-08 00:29:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 140355 (14h13m13.036197215s) [upload=14641]: Starting upload Aug 7 23:52:51.292216133 (14h13m13.036272985s) [upload=14641]: Done starting upload Aug 7 23:52:51.292307446 (14h13m13.036364307s) [upload=14641]: Uploading segments... Aug 7 23:52:51.292389087 (14h13m13.036445956s) [upload=14641]: Got next segment. Inline: true Aug 7 23:52:51.292394933 (14h13m13.036451793s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:52:51.292456032 (14h13m13.036512883s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:51.292461787 (14h13m13.036518647s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:51.292467676 (14h13m13.036524537s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:51.436915828 (14h13m13.180972678s) [upload=14641]: Done uploading segments... Aug 7 23:52:51.436934458 (14h13m13.18099131s) [upload=14641]: Upload finished. err: <nil> Aug 7 23:53:10.946261153 (14h13m32.690318028s) [upload=14646]: Starting upload Aug 7 23:53:10.946350198 (14h13m32.690407065s) [upload=14646]: Done starting upload Aug 7 23:53:10.946435423 (14h13m32.690492276s) [upload=14646]: Uploading segments... Aug 7 23:53:10.946489255 (14h13m32.690546124s) [upload=14646]: Got next segment. Inline: true Aug 7 23:53:10.946497660 (14h13m32.690554519s) [upload=14646]: Waiting for error group managing segments... Aug 7 23:53:10.946542678 (14h13m32.690599529s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:10.946548267 (14h13m32.690605128s) [upload=14646]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:10.946553792 (14h13m32.690610652s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:14.554651436 (14h13m36.298708309s) [upload=14646]: Done uploading segments... Aug 7 23:53:14.554675183 (14h13m36.298732045s) [upload=14646]: Upload finished. err: <nil> Aug 7 23:53:53.584502029 (14h14m15.328558897s) [upload=14658]: Starting upload Aug 7 23:53:53.584599757 (14h14m15.328656616s) [upload=14658]: Done starting upload Aug 7 23:53:53.584702364 (14h14m15.328759218s) [upload=14658]: Uploading segments... Aug 7 23:53:53.584798610 (14h14m15.32885548s) [upload=14658]: Got next segment. Inline: true Aug 7 23:53:53.584806594 (14h14m15.328863461s) [upload=14658]: Waiting for error group managing segments... Aug 7 23:53:53.584875193 (14h14m15.328932062s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:53.584882324 (14h14m15.328939193s) [upload=14658]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:53.584901657 (14h14m15.328958528s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:23.318821682 (14h14m45.062878542s) [upload=14658]: Done uploading segments... Aug 7 23:54:23.318863033 (14h14m45.062919884s) [upload=14658]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:36.775209359 (14h14m58.519266209s) [upload=14671]: Starting upload Aug 7 23:54:36.775292235 (14h14m58.519349112s) [upload=14671]: Done starting upload Aug 7 23:54:36.775412031 (14h14m58.519468883s) [upload=14671]: Uploading segments... Aug 7 23:54:36.775480767 (14h14m58.519537636s) [upload=14671]: Got next segment. Inline: true Aug 7 23:54:36.775487320 (14h14m58.519544181s) [upload=14671]: Waiting for error group managing segments... Aug 7 23:54:36.775539752 (14h14m58.519596612s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:36.775545893 (14h14m58.519602753s) [upload=14671]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:36.775549619 (14h14m58.519606479s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:36.926276427 (14h14m58.670333277s) [upload=14671]: Done uploading segments... Aug 7 23:54:36.926295206 (14h14m58.670352057s) [upload=14671]: Upload finished. err: <nil> Aug 7 23:54:57.675347843 (14h15m19.419404694s) [upload=14676]: Starting upload Aug 7 23:54:57.675412352 (14h15m19.419469212s) [upload=14676]: Done starting upload Aug 7 23:54:57.675475670 (14h15m19.41953252s) [upload=14676]: Uploading segments... Aug 7 23:54:57.675516336 (14h15m19.419573205s) [upload=14676]: Got next segment. Inline: true Aug 7 23:54:57.675521071 (14h15m19.419577931s) [upload=14676]: Waiting for error group managing segments... Aug 7 23:54:57.675565512 (14h15m19.419622375s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:57.675574255 (14h15m19.419631202s) [upload=14676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:57.675580694 (14h15m19.419637563s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:27.407552485 (14h15m49.151609344s) [upload=14676]: Done uploading segments... Aug 7 23:55:27.407592978 (14h15m49.151649829s) [upload=14676]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:06.851646232 (14h16m28.595703082s) [upload=14696]: Starting upload Aug 7 23:56:06.851717330 (14h16m28.595774191s) [upload=14696]: Done starting upload Aug 7 23:56:06.851786008 (14h16m28.59584286s) [upload=14696]: Uploading segments... Aug 7 23:56:06.851834232 (14h16m28.595891101s) [upload=14696]: Got next segment. Inline: true Aug 7 23:56:06.851839675 (14h16m28.595896535s) [upload=14696]: Waiting for error group managing segments... Aug 7 23:56:06.851896275 (14h16m28.595953125s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:06.851901659 (14h16m28.59595852s) [upload=14696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:06.851905505 (14h16m28.595962364s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:07.006202008 (14h16m28.750258868s) [upload=14696]: Done uploading segments... Aug 7 23:56:07.006222000 (14h16m28.750278861s) [upload=14696]: Upload finished. err: <nil> Aug 7 23:56:59.020763574 (14h17m20.764820421s) [upload=14712]: Starting upload Aug 7 23:56:59.020858955 (14h17m20.764915806s) [upload=14712]: Done starting upload Aug 7 23:56:59.020935444 (14h17m20.764992295s) [upload=14712]: Uploading segments... Aug 7 23:56:59.021007501 (14h17m20.76506437s) [upload=14712]: Got next segment. Inline: true Aug 7 23:56:59.021013631 (14h17m20.765070491s) [upload=14712]: Waiting for error group managing segments... Aug 7 23:56:59.021070471 (14h17m20.76512733s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:59.021076966 (14h17m20.765133821s) [upload=14712]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:59.021081305 (14h17m20.76513816s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:59.154820708 (14h17m20.898877569s) [upload=14712]: Done uploading segments... Aug 7 23:56:59.154839835 (14h17m20.898896685s) [upload=14712]: Upload finished. err: <nil> Aug 7 23:58:10.963677657 (14h18m32.707734508s) [upload=14728]: Starting upload Aug 7 23:58:10.963770332 (14h18m32.70782718s) [upload=14728]: Done starting upload Aug 7 23:58:10.963851725 (14h18m32.707908586s) [upload=14728]: Uploading segments... Aug 7 23:58:10.963901559 (14h18m32.707958431s) [upload=14728]: Got next segment. Inline: true Aug 7 23:58:10.963907663 (14h18m32.707964525s) [upload=14728]: Waiting for error group managing segments... Aug 7 23:58:10.963960167 (14h18m32.708017032s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:10.963966034 (14h18m32.708022896s) [upload=14728]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:10.963969944 (14h18m32.708026806s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.106069715 (14h18m32.850126571s) [upload=14728]: Done uploading segments... Aug 7 23:58:11.106112099 (14h18m32.850168961s) [upload=14728]: Upload finished. err: <nil> Aug 7 23:59:19.078128070 (14h19m40.822184931s) [upload=14749]: Starting upload Aug 7 23:59:19.078217499 (14h19m40.822274359s) [upload=14749]: Done starting upload Aug 7 23:59:19.078291040 (14h19m40.822347895s) [upload=14749]: Uploading segments... Aug 7 23:59:19.078333863 (14h19m40.822390732s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:19.078339134 (14h19m40.822395984s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:19.078391868 (14h19m40.822448717s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:19.078397213 (14h19m40.822454064s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:19.078401054 (14h19m40.822457905s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:48.808547234 (14h20m10.552604081s) [upload=14749]: Done uploading segments... Aug 7 23:59:48.808568563 (14h20m10.552625409s) [upload=14749]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:20.896255037 (14h20m42.640311905s) [upload=14765]: Starting upload Aug 8 00:00:20.896381648 (14h20m42.640438517s) [upload=14765]: Done starting upload Aug 8 00:00:20.896532416 (14h20m42.640589282s) [upload=14765]: Uploading segments... Aug 8 00:00:20.896610121 (14h20m42.640666994s) [upload=14765]: Got next segment. Inline: true Aug 8 00:00:20.896618785 (14h20m42.640675649s) [upload=14765]: Waiting for error group managing segments... Aug 8 00:00:20.896682486 (14h20m42.640739601s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:20.896690306 (14h20m42.640747187s) [upload=14765]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:20.896694946 (14h20m42.640751814s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:50.607106781 (14h21m12.351163663s) [upload=14765]: Done uploading segments... Aug 8 00:00:50.607134684 (14h21m12.351191558s) [upload=14765]: Upload finished. err: metaclient: context deadline exceeded Duration: 879.706646ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........XX.X.X...XX......................................] duration: 2.530761775s FAIL (2025-08-08 00:29:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 6]: Uploading piece... Aug 7 23:57:10.893816491 (14h17m32.637873354s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.894285524 (14h17m32.638342391s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895348932 (14h17m32.639405787s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895932061 (14h17m32.639988921s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.896545320 (14h17m32.640602184s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897117472 (14h17m32.641174341s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897621325 (14h17m32.641678192s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898101338 (14h17m32.642158204s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898693607 (14h17m32.642750474s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.900512475 (14h17m32.644569335s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.902337804 (14h17m32.646394669s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.904215667 (14h17m32.648272537s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.906034481 (14h17m32.650091365s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.907857426 (14h17m32.651914295s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.908666188 (14h17m32.652723069s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:11.510238622 (14h17m33.254295484s) [upload=14716]: Next returned nil segment Aug 7 23:57:11.510261464 (14h17m33.254318324s) [upload=14716]: Waiting for error group managing segments... Aug 7 23:57:11.657094388 (14h17m33.401151236s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.674408374 (14h17m33.418465234s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.681203586 (14h17m33.425260449s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.682780850 (14h17m33.4268377s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.684042315 (14h17m33.428099176s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.719658547 (14h17m33.463715401s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.730348391 (14h17m33.474405242s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.748264399 (14h17m33.492321249s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.769764028 (14h17m33.513820878s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.774177874 (14h17m33.518234733s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.789864472 (14h17m33.533921326s) [noise=true , address=45.91.116.13:4153 , piece_id=WBFFQT5I6ZR5ZKUJRFULNOJZ455KIJMA5EEATDDOCVFUKRD67NFA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.799519597 (14h17m33.543576447s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800753821 (14h17m33.544810667s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800785983 (14h17m33.544842842s) [seg_pos={0 0}, upload=14716]: Segment reached optimal threshold of 13 pieces. Aug 7 23:57:11.801026052 (14h17m33.545082898s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801098020 (14h17m33.545154867s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801174944 (14h17m33.54523179s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection Aug 7 23:57:11.801184926 (14h17m33.545241773s) [seg_pos={0 0}, upload=14716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:57:11.801257096 (14h17m33.545313956s) [upload=14716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:11.801261009 (14h17m33.545317869s) [upload=14716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:11.942784613 (14h17m33.686841513s) [upload=14716]: Done uploading segments... Aug 7 23:57:11.942815424 (14h17m33.686872279s) [upload=14716]: Upload finished. err: <nil> Aug 7 23:58:20.561350310 (14h18m42.305407184s) [upload=14732]: Starting upload Aug 7 23:58:20.561483402 (14h18m42.30554027s) [upload=14732]: Done starting upload Aug 7 23:58:20.561650297 (14h18m42.305707159s) [upload=14732]: Uploading segments... Aug 7 23:58:20.562196341 (14h18m42.306253212s) [upload=14732]: Got next segment. Inline: false Aug 7 23:58:20.562334166 (14h18m42.306391031s) [upload=14732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:20.562343255 (14h18m42.306400125s) [upload=14732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:50.281195862 (14h19m12.025252712s) [upload=14732]: Next segment err: context deadline exceeded Aug 7 23:58:50.281383705 (14h19m12.025440555s) [upload=14732]: Done uploading segments... Aug 7 23:58:50.281405600 (14h19m12.02546245s) [upload=14732]: Upload finished. err: context deadline exceeded Aug 7 23:58:50.716777496 (14h19m12.460834346s) [upload=14742]: Starting upload Aug 7 23:58:50.716857905 (14h19m12.460914751s) [upload=14742]: Done starting upload Aug 7 23:58:50.716999972 (14h19m12.461056825s) [upload=14742]: Uploading segments... Aug 7 23:58:50.717186124 (14h19m12.461242976s) [upload=14742]: Got next segment. Inline: false Aug 7 23:58:50.717245601 (14h19m12.461302466s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:50.717252185 (14h19m12.461309053s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:20.436529435 (14h19m42.180586305s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:20.436734882 (14h19m42.180791733s) [upload=14742]: Done uploading segments... Aug 7 23:59:20.436756200 (14h19m42.180813054s) [upload=14742]: Upload finished. err: context deadline exceeded Duration: 2.530794909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [...........XX..XX.X.X.......................................] duration: 876.986144ms FAIL (2025-08-08 00:29:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0]: Starting upload Aug 7 23:53:17.153036870 (14h13m38.89709373s) [upload=14650]: Done starting upload Aug 7 23:53:17.153154678 (14h13m38.897211544s) [upload=14650]: Uploading segments... Aug 7 23:53:17.153233089 (14h13m38.897289957s) [upload=14650]: Got next segment. Inline: true Aug 7 23:53:17.153240256 (14h13m38.897297124s) [upload=14650]: Waiting for error group managing segments... Aug 7 23:53:17.153318892 (14h13m38.897375757s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:17.153325816 (14h13m38.89738268s) [upload=14650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:17.153330453 (14h13m38.897387314s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:16.875757982 (14h14m38.619814845s) [upload=14650]: Done uploading segments... Aug 7 23:54:16.875781849 (14h14m38.619838701s) [upload=14650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:24.158448119 (14h14m45.90250498s) [upload=14667]: Starting upload Aug 7 23:54:24.158525946 (14h14m45.902582806s) [upload=14667]: Done starting upload Aug 7 23:54:24.158598972 (14h14m45.902655824s) [upload=14667]: Uploading segments... Aug 7 23:54:24.158654148 (14h14m45.902711017s) [upload=14667]: Got next segment. Inline: true Aug 7 23:54:24.158659995 (14h14m45.902716856s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:54:24.158714896 (14h14m45.902771747s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:24.158720357 (14h14m45.902777209s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:24.158724105 (14h14m45.902780965s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:23.888400190 (14h15m45.63245704s) [upload=14667]: Done uploading segments... Aug 7 23:55:23.888419087 (14h15m45.632475938s) [upload=14667]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:24.297146309 (14h15m46.041203174s) [upload=14687]: Starting upload Aug 7 23:55:24.297232144 (14h15m46.04128901s) [upload=14687]: Done starting upload Aug 7 23:55:24.297315772 (14h15m46.041372619s) [upload=14687]: Uploading segments... Aug 7 23:55:24.297396238 (14h15m46.041453151s) [upload=14687]: Got next segment. Inline: true Aug 7 23:55:24.297403729 (14h15m46.04146059s) [upload=14687]: Waiting for error group managing segments... Aug 7 23:55:24.297444116 (14h15m46.041500976s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:24.297450971 (14h15m46.041507831s) [upload=14687]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:24.297453883 (14h15m46.041510749s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:24.439126204 (14h15m46.183183055s) [upload=14687]: Done uploading segments... Aug 7 23:55:24.439141121 (14h15m46.183197972s) [upload=14687]: Upload finished. err: <nil> Aug 7 23:56:03.068174127 (14h16m24.812230977s) [upload=14695]: Starting upload Aug 7 23:56:03.068258755 (14h16m24.812315617s) [upload=14695]: Done starting upload Aug 7 23:56:03.068368961 (14h16m24.812425812s) [upload=14695]: Uploading segments... Aug 7 23:56:03.068437851 (14h16m24.812494719s) [upload=14695]: Got next segment. Inline: true Aug 7 23:56:03.068445058 (14h16m24.812501927s) [upload=14695]: Waiting for error group managing segments... Aug 7 23:56:03.068516175 (14h16m24.812573049s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:03.068523275 (14h16m24.812580137s) [upload=14695]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:03.068527513 (14h16m24.812584488s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:03.370378590 (14h16m25.114435452s) [upload=14695]: Done uploading segments... Aug 7 23:56:03.370401742 (14h16m25.114458656s) [upload=14695]: Upload finished. err: <nil> Aug 7 23:56:41.636734874 (14h17m3.380791742s) [upload=14707]: Starting upload Aug 7 23:56:41.636844337 (14h17m3.380901201s) [upload=14707]: Done starting upload Aug 7 23:56:41.636938361 (14h17m3.380995223s) [upload=14707]: Uploading segments... Aug 7 23:56:41.637004566 (14h17m3.381061436s) [upload=14707]: Got next segment. Inline: true Aug 7 23:56:41.637011720 (14h17m3.381068581s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:56:41.637073380 (14h17m3.381130242s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:41.637079684 (14h17m3.381136546s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:41.637083892 (14h17m3.381140753s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:41.369074470 (14h18m3.113131318s) [upload=14707]: Done uploading segments... Aug 7 23:57:41.369096313 (14h18m3.11315316s) [upload=14707]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:41.785857243 (14h18m3.529914104s) [upload=14726]: Starting upload Aug 7 23:57:41.785973295 (14h18m3.530030155s) [upload=14726]: Done starting upload Aug 7 23:57:41.786053418 (14h18m3.530110268s) [upload=14726]: Uploading segments... Aug 7 23:57:41.786115407 (14h18m3.530172273s) [upload=14726]: Got next segment. Inline: true Aug 7 23:57:41.786121268 (14h18m3.530178127s) [upload=14726]: Waiting for error group managing segments... Aug 7 23:57:41.786162041 (14h18m3.530218901s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:41.786167247 (14h18m3.530224108s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:41.786170583 (14h18m3.530227443s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.518215592 (14h19m3.262272452s) [upload=14726]: Done uploading segments... Aug 7 23:58:41.518237426 (14h19m3.262294275s) [upload=14726]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:49.080161447 (14h19m10.824218292s) [upload=14741]: Starting upload Aug 7 23:58:49.080236560 (14h19m10.824293407s) [upload=14741]: Done starting upload Aug 7 23:58:49.080313897 (14h19m10.824370744s) [upload=14741]: Uploading segments... Aug 7 23:58:49.080371825 (14h19m10.824428691s) [upload=14741]: Got next segment. Inline: true Aug 7 23:58:49.080376725 (14h19m10.824433586s) [upload=14741]: Waiting for error group managing segments... Aug 7 23:58:49.080418334 (14h19m10.824475197s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:49.080422560 (14h19m10.824479419s) [upload=14741]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:49.080438323 (14h19m10.824495186s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:49.220444289 (14h19m10.964501139s) [upload=14741]: Done uploading segments... Aug 7 23:58:49.220462660 (14h19m10.96451951s) [upload=14741]: Upload finished. err: <nil> Aug 7 23:59:26.271206280 (14h19m48.015263132s) [upload=14752]: Starting upload Aug 7 23:59:26.271300842 (14h19m48.015357706s) [upload=14752]: Done starting upload Aug 7 23:59:26.271409542 (14h19m48.015466389s) [upload=14752]: Uploading segments... Aug 7 23:59:26.271471969 (14h19m48.015528839s) [upload=14752]: Got next segment. Inline: true Aug 7 23:59:26.271478422 (14h19m48.015535286s) [upload=14752]: Waiting for error group managing segments... Aug 7 23:59:26.271524304 (14h19m48.015581168s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.271531803 (14h19m48.01558867s) [upload=14752]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:26.271535740 (14h19m48.015592604s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.006104454 (14h20m47.750161323s) [upload=14752]: Done uploading segments... Aug 8 00:00:26.006140966 (14h20m47.750197816s) [upload=14752]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:26.408809053 (14h20m48.152865927s) [upload=14766]: Starting upload Aug 8 00:00:26.408886561 (14h20m48.152943434s) [upload=14766]: Done starting upload Aug 8 00:00:26.408975679 (14h20m48.15303253s) [upload=14766]: Uploading segments... Aug 8 00:00:26.409028539 (14h20m48.153085476s) [upload=14766]: Got next segment. Inline: true Aug 8 00:00:26.409034850 (14h20m48.153091712s) [upload=14766]: Waiting for error group managing segments... Aug 8 00:00:26.409085190 (14h20m48.153142042s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:26.409090930 (14h20m48.153147793s) [upload=14766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:26.409094742 (14h20m48.153151605s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.549853950 (14h20m48.2939108s) [upload=14766]: Done uploading segments... Aug 8 00:00:26.549878419 (14h20m48.293935265s) [upload=14766]: Upload finished. err: <nil> Aug 8 00:01:08.274172938 (14h21m30.018229799s) [upload=14780]: Starting upload Aug 8 00:01:08.274248443 (14h21m30.018305302s) [upload=14780]: Done starting upload Aug 8 00:01:08.274326739 (14h21m30.018383585s) [upload=14780]: Uploading segments... Aug 8 00:01:08.274380689 (14h21m30.018437549s) [upload=14780]: Got next segment. Inline: true Aug 8 00:01:08.274385704 (14h21m30.018442565s) [upload=14780]: Waiting for error group managing segments... Aug 8 00:01:08.274437335 (14h21m30.018494182s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:08.274442821 (14h21m30.018499667s) [upload=14780]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:08.274445609 (14h21m30.018502454s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:08.005343335 (14h22m29.749400197s) [upload=14780]: Done uploading segments... Aug 8 00:02:08.005365121 (14h22m29.749421982s) [upload=14780]: Upload finished. err: metaclient: context deadline exceeded Duration: 877.094603ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 109.57236ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-08 00:29:16 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 109.621251ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............XX.X.X...XX.....................................] duration: 2.444919774s FAIL (2025-08-08 00:28:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 6]: Uploading piece... Aug 7 23:57:10.893816491 (14h17m32.637873354s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.894285524 (14h17m32.638342391s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895348932 (14h17m32.639405787s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895932061 (14h17m32.639988921s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.896545320 (14h17m32.640602184s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897117472 (14h17m32.641174341s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897621325 (14h17m32.641678192s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898101338 (14h17m32.642158204s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898693607 (14h17m32.642750474s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.900512475 (14h17m32.644569335s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.902337804 (14h17m32.646394669s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.904215667 (14h17m32.648272537s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.906034481 (14h17m32.650091365s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.907857426 (14h17m32.651914295s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.908666188 (14h17m32.652723069s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:11.510238622 (14h17m33.254295484s) [upload=14716]: Next returned nil segment Aug 7 23:57:11.510261464 (14h17m33.254318324s) [upload=14716]: Waiting for error group managing segments... Aug 7 23:57:11.657094388 (14h17m33.401151236s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.674408374 (14h17m33.418465234s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.681203586 (14h17m33.425260449s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.682780850 (14h17m33.4268377s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.684042315 (14h17m33.428099176s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.719658547 (14h17m33.463715401s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.730348391 (14h17m33.474405242s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.748264399 (14h17m33.492321249s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.769764028 (14h17m33.513820878s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.774177874 (14h17m33.518234733s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.789864472 (14h17m33.533921326s) [noise=true , address=45.91.116.13:4153 , piece_id=WBFFQT5I6ZR5ZKUJRFULNOJZ455KIJMA5EEATDDOCVFUKRD67NFA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.799519597 (14h17m33.543576447s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800753821 (14h17m33.544810667s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800785983 (14h17m33.544842842s) [seg_pos={0 0}, upload=14716]: Segment reached optimal threshold of 13 pieces. Aug 7 23:57:11.801026052 (14h17m33.545082898s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801098020 (14h17m33.545154867s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801174944 (14h17m33.54523179s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection Aug 7 23:57:11.801184926 (14h17m33.545241773s) [seg_pos={0 0}, upload=14716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:57:11.801257096 (14h17m33.545313956s) [upload=14716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:11.801261009 (14h17m33.545317869s) [upload=14716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:11.942784613 (14h17m33.686841513s) [upload=14716]: Done uploading segments... Aug 7 23:57:11.942815424 (14h17m33.686872279s) [upload=14716]: Upload finished. err: <nil> Aug 7 23:58:20.561350310 (14h18m42.305407184s) [upload=14732]: Starting upload Aug 7 23:58:20.561483402 (14h18m42.30554027s) [upload=14732]: Done starting upload Aug 7 23:58:20.561650297 (14h18m42.305707159s) [upload=14732]: Uploading segments... Aug 7 23:58:20.562196341 (14h18m42.306253212s) [upload=14732]: Got next segment. Inline: false Aug 7 23:58:20.562334166 (14h18m42.306391031s) [upload=14732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:20.562343255 (14h18m42.306400125s) [upload=14732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:50.281195862 (14h19m12.025252712s) [upload=14732]: Next segment err: context deadline exceeded Aug 7 23:58:50.281383705 (14h19m12.025440555s) [upload=14732]: Done uploading segments... Aug 7 23:58:50.281405600 (14h19m12.02546245s) [upload=14732]: Upload finished. err: context deadline exceeded Aug 7 23:58:50.716777496 (14h19m12.460834346s) [upload=14742]: Starting upload Aug 7 23:58:50.716857905 (14h19m12.460914751s) [upload=14742]: Done starting upload Aug 7 23:58:50.716999972 (14h19m12.461056825s) [upload=14742]: Uploading segments... Aug 7 23:58:50.717186124 (14h19m12.461242976s) [upload=14742]: Got next segment. Inline: false Aug 7 23:58:50.717245601 (14h19m12.461302466s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:50.717252185 (14h19m12.461309053s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:20.436529435 (14h19m42.180586305s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:20.436734882 (14h19m42.180791733s) [upload=14742]: Done uploading segments... Aug 7 23:59:20.436756200 (14h19m42.180813054s) [upload=14742]: Upload finished. err: context deadline exceeded Duration: 2.444946863s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X.XX...XX.X..X..................................] duration: 2.642020835s FAIL (2025-08-08 00:28:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.642079569s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [......X.XXX.X.X.....X.......................................] duration: 918.093703ms FAIL (2025-08-08 00:28:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 918.127834ms core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.93381454s FAIL (2025-08-08 00:28:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.933854644s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..............X...................] duration: 2.9459271s FAIL (2025-08-08 00:28:44 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== oad Aug 8 00:12:40.688890451 (14h33m2.432947302s) [upload=14976]: Uploading segments... Aug 8 00:12:40.689315108 (14h33m2.433371968s) [upload=14976]: Got next segment. Inline: false Aug 8 00:12:40.689365721 (14h33m2.433422571s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:12:40.689371637 (14h33m2.433428498s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:12:40.938712718 (14h33m2.682769583s) [seg_pos={0 0}, upload=14976]: Begin upload segment... Aug 8 00:12:40.938780540 (14h33m2.68283739s) [seg_pos={0 0}, upload=14976]: Done begin upload segment. Aug 8 00:12:40.938826511 (14h33m2.682883372s) [noise=true , address=104.36.82.11:4122 , piece_id=T4ATL4PZUGFLXWE47UGHDXX3ID2LEJKQRGQQMF2G4J42TET2JVYQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.938837427 (14h33m2.682894292s) [noise=true , address=64.49.13.169:4715 , piece_id=TKX4G5GWLDNDPR66UADVNNWV2TAFGCVEMWZAU4GROJXFX3Q537TA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939397428 (14h33m2.683454296s) [noise=true , address=104.36.82.44:4881 , piece_id=TS5LURERTPPJMH3KOK3ABNKEWSWOF7IUNNKP5DONPQTXKIF4S55Q, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939417193 (14h33m2.683474059s) [noise=true , address=104.36.82.5:4859 , piece_id=FHBE3H7NTIY3PHCICRNYDGY7MEILS27N4O5DLKQQWM5UXELX3EWQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939726153 (14h33m2.683783022s) [noise=true , address=104.36.82.45:4937 , piece_id=4Z6T6K7EP3LS3XYSTOARORNVPGKX6HYIHV33I665O5FI4OPGOXNA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939786076 (14h33m2.683842943s) [noise=true , address=104.36.82.23:4453 , piece_id=V7D5MY7E63SEALXMVDJOS26NYQDQAFJ4SHUISCK6TPKDQKKI4Z3Q, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940057147 (14h33m2.68411401s) [noise=true , address=104.36.82.22:4537 , piece_id=3Q4MNZBN5UPYVL4BC673XQVPMLW3MYTY2CNLKFE7NW6ZWSFHAWPQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940208252 (14h33m2.684265105s) [noise=true , address=104.36.82.59:4313 , piece_id=LIWITE4IJVNHLO2DFFDI2M4D67UAHLM223EF6FAXGNGWIIMGH7FA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940408469 (14h33m2.68446533s) [noise=true , address=64.49.13.154:4930 , piece_id=OGQJV2HMKOE6IPET6A6JRENBWMGJ6NWSGL7Q55653IB6QBAUARQA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940742041 (14h33m2.684798902s) [noise=true , address=45.91.116.7:4034 , piece_id=XJQV5V55F3KLAF5MZZMWZN7HE6TRHQ2V6BDDMKOWMDMKEBAP6UZQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941021709 (14h33m2.685078579s) [noise=true , address=104.36.82.37:4671 , piece_id=HYA6VNL24EERU27SJLYO7TCV7DQIIH7Q4LXAEA5BXSHBTIXCHKDQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941299121 (14h33m2.685355986s) [noise=true , address=104.36.82.43:4855 , piece_id=AMNGI7UV2PCEQ2U4LAOXIFT4T3WXPD7YHLR77VR7MSSQGHESY5EA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941588088 (14h33m2.685644951s) [noise=true , address=104.36.82.19:4401 , piece_id=NACQ6PBCDIHNJAQIFQUVYAPU2HY5C4ZDW6QMJBGY3E45TNWHFQNQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941637056 (14h33m2.685693925s) [noise=true , address=104.36.82.61:4404 , piece_id=F5YUVKGDN4MPHJKOLR5DRVABOU6326LGWAJXKL4RCWZI35TLMXKQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941950914 (14h33m2.686007788s) [noise=true , address=79.127.201.215:4351 , piece_id=DR37C77MS5YTW7DK7XOJQGMEXWOSUKOD76SZY7TXLIB5RLXLEVMQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.942345807 (14h33m2.686402678s) [noise=true , address=104.36.82.55:4206 , piece_id=FEFQ3F2EFHMZI53KUIAGHESBATXL5S2EOAARLOHJ2YDSMGVLZRQQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.942560802 (14h33m2.686617667s) [noise=true , address=104.36.82.12:4158 , piece_id=SCBLDCSKC6HTOUIUA43BA4452TPHEIVKCMOSVHR42YXVUWYUFPRA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.943202713 (14h33m2.687259576s) [noise=true , address=193.149.164.143:4693 , piece_id=P5YP7XDPKE2VYGTIJMAD6TV2LAPXJRN47E3PDPDDLVNI73VJGUBA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:41.525441834 (14h33m3.269498698s) [upload=14976]: Next returned nil segment Aug 8 00:12:41.525462664 (14h33m3.269519531s) [upload=14976]: Waiting for error group managing segments... Aug 8 00:12:41.699976519 (14h33m3.444033384s) [noise=true , address=104.36.82.19:4401 , piece_id=NACQ6PBCDIHNJAQIFQUVYAPU2HY5C4ZDW6QMJBGY3E45TNWHFQNQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.700944219 (14h33m3.445001137s) [noise=true , address=104.36.82.23:4453 , piece_id=V7D5MY7E63SEALXMVDJOS26NYQDQAFJ4SHUISCK6TPKDQKKI4Z3Q, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.702379298 (14h33m3.446436158s) [noise=true , address=104.36.82.37:4671 , piece_id=HYA6VNL24EERU27SJLYO7TCV7DQIIH7Q4LXAEA5BXSHBTIXCHKDQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.710467855 (14h33m3.454524716s) [noise=true , address=104.36.82.55:4206 , piece_id=FEFQ3F2EFHMZI53KUIAGHESBATXL5S2EOAARLOHJ2YDSMGVLZRQQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.716048399 (14h33m3.46010525s) [noise=true , address=104.36.82.22:4537 , piece_id=3Q4MNZBN5UPYVL4BC673XQVPMLW3MYTY2CNLKFE7NW6ZWSFHAWPQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.769601981 (14h33m3.513658832s) [noise=true , address=104.36.82.44:4881 , piece_id=TS5LURERTPPJMH3KOK3ABNKEWSWOF7IUNNKP5DONPQTXKIF4S55Q, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.774083597 (14h33m3.518140448s) [noise=true , address=104.36.82.5:4859 , piece_id=FHBE3H7NTIY3PHCICRNYDGY7MEILS27N4O5DLKQQWM5UXELX3EWQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.790244473 (14h33m3.534301324s) [noise=true , address=45.91.116.7:4034 , piece_id=XJQV5V55F3KLAF5MZZMWZN7HE6TRHQ2V6BDDMKOWMDMKEBAP6UZQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.790597987 (14h33m3.534654837s) [noise=true , address=104.36.82.61:4404 , piece_id=F5YUVKGDN4MPHJKOLR5DRVABOU6326LGWAJXKL4RCWZI35TLMXKQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.814650455 (14h33m3.558707307s) [noise=true , address=104.36.82.59:4313 , piece_id=LIWITE4IJVNHLO2DFFDI2M4D67UAHLM223EF6FAXGNGWIIMGH7FA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.916929838 (14h33m3.660986699s) [noise=true , address=104.36.82.12:4158 , piece_id=SCBLDCSKC6HTOUIUA43BA4452TPHEIVKCMOSVHR42YXVUWYUFPRA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.946542933 (14h33m3.690599791s) [noise=true , address=79.127.201.215:4351 , piece_id=DR37C77MS5YTW7DK7XOJQGMEXWOSUKOD76SZY7TXLIB5RLXLEVMQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.014532160 (14h33m3.75858901s) [noise=true , address=104.36.82.43:4855 , piece_id=AMNGI7UV2PCEQ2U4LAOXIFT4T3WXPD7YHLR77VR7MSSQGHESY5EA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.157107757 (14h33m3.901164607s) [noise=true , address=104.36.82.11:4122 , piece_id=T4ATL4PZUGFLXWE47UGHDXX3ID2LEJKQRGQQMF2G4J42TET2JVYQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.157146638 (14h33m3.901203502s) [seg_pos={0 0}, upload=14976]: Segment reached optimal threshold of 14 pieces. Aug 8 00:12:42.157376876 (14h33m3.901433737s) [noise=true , address=64.49.13.169:4715 , piece_id=TKX4G5GWLDNDPR66UADVNNWV2TAFGCVEMWZAU4GROJXFX3Q537TA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157574941 (14h33m3.901631791s) [noise=true , address=193.149.164.143:4693 , piece_id=P5YP7XDPKE2VYGTIJMAD6TV2LAPXJRN47E3PDPDDLVNI73VJGUBA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LHoCE9CnM32tbMNqAB7HpmArSVLTycv9o6kSSFNUJsoDeERkQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157631621 (14h33m3.901688486s) [noise=true , address=64.49.13.154:4930 , piece_id=OGQJV2HMKOE6IPET6A6JRENBWMGJ6NWSGL7Q55653IB6QBAUARQA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kNK8y5et7MiiWNDT6iaC8YBAk3PX6WX1XSuU4QMSFBp3KymBb): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157680271 (14h33m3.901737121s) [noise=true , address=104.36.82.45:4937 , piece_id=4Z6T6K7EP3LS3XYSTOARORNVPGKX6HYIHV33I665O5FI4OPGOXNA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection; write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection Aug 8 00:12:42.157698358 (14h33m3.901755219s) [seg_pos={0 0}, upload=14976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:12:42.157816415 (14h33m3.901873289s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:12:42.157839617 (14h33m3.901896479s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:12:42.510566001 (14h33m4.254622851s) [upload=14976]: Done uploading segments... Aug 8 00:12:42.510584893 (14h33m4.254641743s) [upload=14976]: Upload finished. err: <nil> Duration: 2.945956689s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 105.445228ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-08 00:28:41 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 105.473836ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............XXX.X.X.......................................] duration: 2.999762205s FAIL (2025-08-08 00:28:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.99978736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................XXX..XX.....................................] duration: 2.900524868s FAIL (2025-08-08 00:28:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 807513521 (14h17m32.551570367s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:55.062716703 (14h18m16.806773554s) [seg_pos={0 0}, upload=14717]: Begin upload segment... Aug 7 23:57:55.062772247 (14h18m16.806829106s) [seg_pos={0 0}, upload=14717]: Done begin upload segment. Aug 7 23:57:55.062817614 (14h18m16.806874477s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.062833275 (14h18m16.806890146s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063321348 (14h18m16.807378211s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063625241 (14h18m16.8076821s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.064591461 (14h18m16.808648334s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065249737 (14h18m16.809306608s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065844042 (14h18m16.809900909s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.066738904 (14h18m16.810795771s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067295868 (14h18m16.81135274s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067868039 (14h18m16.811924903s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068378475 (14h18m16.812435346s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068951556 (14h18m16.81300842s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069355853 (14h18m16.813412717s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069707384 (14h18m16.813764245s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070103904 (14h18m16.814160763s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070434322 (14h18m16.814491182s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070775335 (14h18m16.814832199s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.072531247 (14h18m16.816588107s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.642344995 (14h18m17.386401842s) [upload=14717]: Next returned nil segment Aug 7 23:57:55.642365727 (14h18m17.38642259s) [upload=14717]: Waiting for error group managing segments... Aug 7 23:57:55.820297514 (14h18m17.564354366s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.832273438 (14h18m17.576330291s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.845342869 (14h18m17.58939972s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.850001106 (14h18m17.594057955s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874147776 (14h18m17.618204625s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874555951 (14h18m17.6186128s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.890084811 (14h18m17.634141672s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.901998021 (14h18m17.646054868s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.930296357 (14h18m17.674353212s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.008745690 (14h18m17.752802541s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.041830348 (14h18m17.785887195s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.086405192 (14h18m17.830462041s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.092724415 (14h18m17.83678127s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095750120 (14h18m17.839806985s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095784368 (14h18m17.839841231s) [seg_pos={0 0}, upload=14717]: Segment reached optimal threshold of 14 pieces. Aug 7 23:57:56.095960472 (14h18m17.840017334s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096160491 (14h18m17.840217355s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096190692 (14h18m17.840247554s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eu7SDhEXZGS6HeAXFZWfxoci2oPdjtYvZ5Bbhvae2neKTxYKo): protocol: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection Aug 7 23:57:56.096286416 (14h18m17.840343264s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w4v4QV7PxhXTwuxbvTQST2buoWNySrziRYTo9Mb8dpENd9XCX): protocol: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection Aug 7 23:57:56.096300170 (14h18m17.840357019s) [seg_pos={0 0}, upload=14717]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:57:56.096419305 (14h18m17.840476164s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:56.096426206 (14h18m17.84048307s) [upload=14717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:30.094636522 (14h22m51.838693376s) [upload=14717]: Done uploading segments... Aug 8 00:02:30.094667105 (14h22m51.838723955s) [upload=14717]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.900563263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration: 1.011482476s FAIL (2025-08-08 00:28:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.011528273s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.......X.XXX.X.X.....X......................................] duration: 899.159397ms FAIL (2025-08-08 00:28:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 899.209442ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............XXX.X.X......................................] duration: 3.064650193s FAIL (2025-08-08 00:28:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.064761479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............XX.X.X...XX....................................] duration: 2.451356987s FAIL (2025-08-08 00:28:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 6]: Uploading piece... Aug 7 23:57:10.893816491 (14h17m32.637873354s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.894285524 (14h17m32.638342391s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895348932 (14h17m32.639405787s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895932061 (14h17m32.639988921s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.896545320 (14h17m32.640602184s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897117472 (14h17m32.641174341s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897621325 (14h17m32.641678192s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898101338 (14h17m32.642158204s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898693607 (14h17m32.642750474s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.900512475 (14h17m32.644569335s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.902337804 (14h17m32.646394669s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.904215667 (14h17m32.648272537s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.906034481 (14h17m32.650091365s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.907857426 (14h17m32.651914295s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.908666188 (14h17m32.652723069s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:11.510238622 (14h17m33.254295484s) [upload=14716]: Next returned nil segment Aug 7 23:57:11.510261464 (14h17m33.254318324s) [upload=14716]: Waiting for error group managing segments... Aug 7 23:57:11.657094388 (14h17m33.401151236s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.674408374 (14h17m33.418465234s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.681203586 (14h17m33.425260449s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.682780850 (14h17m33.4268377s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.684042315 (14h17m33.428099176s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.719658547 (14h17m33.463715401s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.730348391 (14h17m33.474405242s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.748264399 (14h17m33.492321249s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.769764028 (14h17m33.513820878s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.774177874 (14h17m33.518234733s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.789864472 (14h17m33.533921326s) [noise=true , address=45.91.116.13:4153 , piece_id=WBFFQT5I6ZR5ZKUJRFULNOJZ455KIJMA5EEATDDOCVFUKRD67NFA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.799519597 (14h17m33.543576447s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800753821 (14h17m33.544810667s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800785983 (14h17m33.544842842s) [seg_pos={0 0}, upload=14716]: Segment reached optimal threshold of 13 pieces. Aug 7 23:57:11.801026052 (14h17m33.545082898s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801098020 (14h17m33.545154867s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801174944 (14h17m33.54523179s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection Aug 7 23:57:11.801184926 (14h17m33.545241773s) [seg_pos={0 0}, upload=14716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:57:11.801257096 (14h17m33.545313956s) [upload=14716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:11.801261009 (14h17m33.545317869s) [upload=14716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:11.942784613 (14h17m33.686841513s) [upload=14716]: Done uploading segments... Aug 7 23:57:11.942815424 (14h17m33.686872279s) [upload=14716]: Upload finished. err: <nil> Aug 7 23:58:20.561350310 (14h18m42.305407184s) [upload=14732]: Starting upload Aug 7 23:58:20.561483402 (14h18m42.30554027s) [upload=14732]: Done starting upload Aug 7 23:58:20.561650297 (14h18m42.305707159s) [upload=14732]: Uploading segments... Aug 7 23:58:20.562196341 (14h18m42.306253212s) [upload=14732]: Got next segment. Inline: false Aug 7 23:58:20.562334166 (14h18m42.306391031s) [upload=14732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:20.562343255 (14h18m42.306400125s) [upload=14732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:50.281195862 (14h19m12.025252712s) [upload=14732]: Next segment err: context deadline exceeded Aug 7 23:58:50.281383705 (14h19m12.025440555s) [upload=14732]: Done uploading segments... Aug 7 23:58:50.281405600 (14h19m12.02546245s) [upload=14732]: Upload finished. err: context deadline exceeded Aug 7 23:58:50.716777496 (14h19m12.460834346s) [upload=14742]: Starting upload Aug 7 23:58:50.716857905 (14h19m12.460914751s) [upload=14742]: Done starting upload Aug 7 23:58:50.716999972 (14h19m12.461056825s) [upload=14742]: Uploading segments... Aug 7 23:58:50.717186124 (14h19m12.461242976s) [upload=14742]: Got next segment. Inline: false Aug 7 23:58:50.717245601 (14h19m12.461302466s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:50.717252185 (14h19m12.461309053s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:20.436529435 (14h19m42.180586305s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:20.436734882 (14h19m42.180791733s) [upload=14742]: Done uploading segments... Aug 7 23:59:20.436756200 (14h19m42.180813054s) [upload=14742]: Upload finished. err: context deadline exceeded Duration: 2.451393066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..............X..................] duration: 3.183062517s FAIL (2025-08-08 00:28:22 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== oad Aug 8 00:12:40.688890451 (14h33m2.432947302s) [upload=14976]: Uploading segments... Aug 8 00:12:40.689315108 (14h33m2.433371968s) [upload=14976]: Got next segment. Inline: false Aug 8 00:12:40.689365721 (14h33m2.433422571s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:12:40.689371637 (14h33m2.433428498s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:12:40.938712718 (14h33m2.682769583s) [seg_pos={0 0}, upload=14976]: Begin upload segment... Aug 8 00:12:40.938780540 (14h33m2.68283739s) [seg_pos={0 0}, upload=14976]: Done begin upload segment. Aug 8 00:12:40.938826511 (14h33m2.682883372s) [noise=true , address=104.36.82.11:4122 , piece_id=T4ATL4PZUGFLXWE47UGHDXX3ID2LEJKQRGQQMF2G4J42TET2JVYQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.938837427 (14h33m2.682894292s) [noise=true , address=64.49.13.169:4715 , piece_id=TKX4G5GWLDNDPR66UADVNNWV2TAFGCVEMWZAU4GROJXFX3Q537TA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939397428 (14h33m2.683454296s) [noise=true , address=104.36.82.44:4881 , piece_id=TS5LURERTPPJMH3KOK3ABNKEWSWOF7IUNNKP5DONPQTXKIF4S55Q, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939417193 (14h33m2.683474059s) [noise=true , address=104.36.82.5:4859 , piece_id=FHBE3H7NTIY3PHCICRNYDGY7MEILS27N4O5DLKQQWM5UXELX3EWQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939726153 (14h33m2.683783022s) [noise=true , address=104.36.82.45:4937 , piece_id=4Z6T6K7EP3LS3XYSTOARORNVPGKX6HYIHV33I665O5FI4OPGOXNA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939786076 (14h33m2.683842943s) [noise=true , address=104.36.82.23:4453 , piece_id=V7D5MY7E63SEALXMVDJOS26NYQDQAFJ4SHUISCK6TPKDQKKI4Z3Q, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940057147 (14h33m2.68411401s) [noise=true , address=104.36.82.22:4537 , piece_id=3Q4MNZBN5UPYVL4BC673XQVPMLW3MYTY2CNLKFE7NW6ZWSFHAWPQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940208252 (14h33m2.684265105s) [noise=true , address=104.36.82.59:4313 , piece_id=LIWITE4IJVNHLO2DFFDI2M4D67UAHLM223EF6FAXGNGWIIMGH7FA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940408469 (14h33m2.68446533s) [noise=true , address=64.49.13.154:4930 , piece_id=OGQJV2HMKOE6IPET6A6JRENBWMGJ6NWSGL7Q55653IB6QBAUARQA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940742041 (14h33m2.684798902s) [noise=true , address=45.91.116.7:4034 , piece_id=XJQV5V55F3KLAF5MZZMWZN7HE6TRHQ2V6BDDMKOWMDMKEBAP6UZQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941021709 (14h33m2.685078579s) [noise=true , address=104.36.82.37:4671 , piece_id=HYA6VNL24EERU27SJLYO7TCV7DQIIH7Q4LXAEA5BXSHBTIXCHKDQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941299121 (14h33m2.685355986s) [noise=true , address=104.36.82.43:4855 , piece_id=AMNGI7UV2PCEQ2U4LAOXIFT4T3WXPD7YHLR77VR7MSSQGHESY5EA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941588088 (14h33m2.685644951s) [noise=true , address=104.36.82.19:4401 , piece_id=NACQ6PBCDIHNJAQIFQUVYAPU2HY5C4ZDW6QMJBGY3E45TNWHFQNQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941637056 (14h33m2.685693925s) [noise=true , address=104.36.82.61:4404 , piece_id=F5YUVKGDN4MPHJKOLR5DRVABOU6326LGWAJXKL4RCWZI35TLMXKQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941950914 (14h33m2.686007788s) [noise=true , address=79.127.201.215:4351 , piece_id=DR37C77MS5YTW7DK7XOJQGMEXWOSUKOD76SZY7TXLIB5RLXLEVMQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.942345807 (14h33m2.686402678s) [noise=true , address=104.36.82.55:4206 , piece_id=FEFQ3F2EFHMZI53KUIAGHESBATXL5S2EOAARLOHJ2YDSMGVLZRQQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.942560802 (14h33m2.686617667s) [noise=true , address=104.36.82.12:4158 , piece_id=SCBLDCSKC6HTOUIUA43BA4452TPHEIVKCMOSVHR42YXVUWYUFPRA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.943202713 (14h33m2.687259576s) [noise=true , address=193.149.164.143:4693 , piece_id=P5YP7XDPKE2VYGTIJMAD6TV2LAPXJRN47E3PDPDDLVNI73VJGUBA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:41.525441834 (14h33m3.269498698s) [upload=14976]: Next returned nil segment Aug 8 00:12:41.525462664 (14h33m3.269519531s) [upload=14976]: Waiting for error group managing segments... Aug 8 00:12:41.699976519 (14h33m3.444033384s) [noise=true , address=104.36.82.19:4401 , piece_id=NACQ6PBCDIHNJAQIFQUVYAPU2HY5C4ZDW6QMJBGY3E45TNWHFQNQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.700944219 (14h33m3.445001137s) [noise=true , address=104.36.82.23:4453 , piece_id=V7D5MY7E63SEALXMVDJOS26NYQDQAFJ4SHUISCK6TPKDQKKI4Z3Q, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.702379298 (14h33m3.446436158s) [noise=true , address=104.36.82.37:4671 , piece_id=HYA6VNL24EERU27SJLYO7TCV7DQIIH7Q4LXAEA5BXSHBTIXCHKDQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.710467855 (14h33m3.454524716s) [noise=true , address=104.36.82.55:4206 , piece_id=FEFQ3F2EFHMZI53KUIAGHESBATXL5S2EOAARLOHJ2YDSMGVLZRQQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.716048399 (14h33m3.46010525s) [noise=true , address=104.36.82.22:4537 , piece_id=3Q4MNZBN5UPYVL4BC673XQVPMLW3MYTY2CNLKFE7NW6ZWSFHAWPQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.769601981 (14h33m3.513658832s) [noise=true , address=104.36.82.44:4881 , piece_id=TS5LURERTPPJMH3KOK3ABNKEWSWOF7IUNNKP5DONPQTXKIF4S55Q, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.774083597 (14h33m3.518140448s) [noise=true , address=104.36.82.5:4859 , piece_id=FHBE3H7NTIY3PHCICRNYDGY7MEILS27N4O5DLKQQWM5UXELX3EWQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.790244473 (14h33m3.534301324s) [noise=true , address=45.91.116.7:4034 , piece_id=XJQV5V55F3KLAF5MZZMWZN7HE6TRHQ2V6BDDMKOWMDMKEBAP6UZQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.790597987 (14h33m3.534654837s) [noise=true , address=104.36.82.61:4404 , piece_id=F5YUVKGDN4MPHJKOLR5DRVABOU6326LGWAJXKL4RCWZI35TLMXKQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.814650455 (14h33m3.558707307s) [noise=true , address=104.36.82.59:4313 , piece_id=LIWITE4IJVNHLO2DFFDI2M4D67UAHLM223EF6FAXGNGWIIMGH7FA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.916929838 (14h33m3.660986699s) [noise=true , address=104.36.82.12:4158 , piece_id=SCBLDCSKC6HTOUIUA43BA4452TPHEIVKCMOSVHR42YXVUWYUFPRA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.946542933 (14h33m3.690599791s) [noise=true , address=79.127.201.215:4351 , piece_id=DR37C77MS5YTW7DK7XOJQGMEXWOSUKOD76SZY7TXLIB5RLXLEVMQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.014532160 (14h33m3.75858901s) [noise=true , address=104.36.82.43:4855 , piece_id=AMNGI7UV2PCEQ2U4LAOXIFT4T3WXPD7YHLR77VR7MSSQGHESY5EA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.157107757 (14h33m3.901164607s) [noise=true , address=104.36.82.11:4122 , piece_id=T4ATL4PZUGFLXWE47UGHDXX3ID2LEJKQRGQQMF2G4J42TET2JVYQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.157146638 (14h33m3.901203502s) [seg_pos={0 0}, upload=14976]: Segment reached optimal threshold of 14 pieces. Aug 8 00:12:42.157376876 (14h33m3.901433737s) [noise=true , address=64.49.13.169:4715 , piece_id=TKX4G5GWLDNDPR66UADVNNWV2TAFGCVEMWZAU4GROJXFX3Q537TA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157574941 (14h33m3.901631791s) [noise=true , address=193.149.164.143:4693 , piece_id=P5YP7XDPKE2VYGTIJMAD6TV2LAPXJRN47E3PDPDDLVNI73VJGUBA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LHoCE9CnM32tbMNqAB7HpmArSVLTycv9o6kSSFNUJsoDeERkQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157631621 (14h33m3.901688486s) [noise=true , address=64.49.13.154:4930 , piece_id=OGQJV2HMKOE6IPET6A6JRENBWMGJ6NWSGL7Q55653IB6QBAUARQA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kNK8y5et7MiiWNDT6iaC8YBAk3PX6WX1XSuU4QMSFBp3KymBb): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157680271 (14h33m3.901737121s) [noise=true , address=104.36.82.45:4937 , piece_id=4Z6T6K7EP3LS3XYSTOARORNVPGKX6HYIHV33I665O5FI4OPGOXNA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection; write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection Aug 8 00:12:42.157698358 (14h33m3.901755219s) [seg_pos={0 0}, upload=14976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:12:42.157816415 (14h33m3.901873289s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:12:42.157839617 (14h33m3.901896479s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:12:42.510566001 (14h33m4.254622851s) [upload=14976]: Done uploading segments... Aug 8 00:12:42.510584893 (14h33m4.254641743s) [upload=14976]: Upload finished. err: <nil> Duration: 3.183106237s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [............XX..XX.X.X......................................] duration: 977.723579ms FAIL (2025-08-08 00:28:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0]: Starting upload Aug 7 23:53:17.153036870 (14h13m38.89709373s) [upload=14650]: Done starting upload Aug 7 23:53:17.153154678 (14h13m38.897211544s) [upload=14650]: Uploading segments... Aug 7 23:53:17.153233089 (14h13m38.897289957s) [upload=14650]: Got next segment. Inline: true Aug 7 23:53:17.153240256 (14h13m38.897297124s) [upload=14650]: Waiting for error group managing segments... Aug 7 23:53:17.153318892 (14h13m38.897375757s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:17.153325816 (14h13m38.89738268s) [upload=14650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:17.153330453 (14h13m38.897387314s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:16.875757982 (14h14m38.619814845s) [upload=14650]: Done uploading segments... Aug 7 23:54:16.875781849 (14h14m38.619838701s) [upload=14650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:24.158448119 (14h14m45.90250498s) [upload=14667]: Starting upload Aug 7 23:54:24.158525946 (14h14m45.902582806s) [upload=14667]: Done starting upload Aug 7 23:54:24.158598972 (14h14m45.902655824s) [upload=14667]: Uploading segments... Aug 7 23:54:24.158654148 (14h14m45.902711017s) [upload=14667]: Got next segment. Inline: true Aug 7 23:54:24.158659995 (14h14m45.902716856s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:54:24.158714896 (14h14m45.902771747s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:24.158720357 (14h14m45.902777209s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:24.158724105 (14h14m45.902780965s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:23.888400190 (14h15m45.63245704s) [upload=14667]: Done uploading segments... Aug 7 23:55:23.888419087 (14h15m45.632475938s) [upload=14667]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:24.297146309 (14h15m46.041203174s) [upload=14687]: Starting upload Aug 7 23:55:24.297232144 (14h15m46.04128901s) [upload=14687]: Done starting upload Aug 7 23:55:24.297315772 (14h15m46.041372619s) [upload=14687]: Uploading segments... Aug 7 23:55:24.297396238 (14h15m46.041453151s) [upload=14687]: Got next segment. Inline: true Aug 7 23:55:24.297403729 (14h15m46.04146059s) [upload=14687]: Waiting for error group managing segments... Aug 7 23:55:24.297444116 (14h15m46.041500976s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:24.297450971 (14h15m46.041507831s) [upload=14687]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:24.297453883 (14h15m46.041510749s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:24.439126204 (14h15m46.183183055s) [upload=14687]: Done uploading segments... Aug 7 23:55:24.439141121 (14h15m46.183197972s) [upload=14687]: Upload finished. err: <nil> Aug 7 23:56:03.068174127 (14h16m24.812230977s) [upload=14695]: Starting upload Aug 7 23:56:03.068258755 (14h16m24.812315617s) [upload=14695]: Done starting upload Aug 7 23:56:03.068368961 (14h16m24.812425812s) [upload=14695]: Uploading segments... Aug 7 23:56:03.068437851 (14h16m24.812494719s) [upload=14695]: Got next segment. Inline: true Aug 7 23:56:03.068445058 (14h16m24.812501927s) [upload=14695]: Waiting for error group managing segments... Aug 7 23:56:03.068516175 (14h16m24.812573049s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:03.068523275 (14h16m24.812580137s) [upload=14695]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:03.068527513 (14h16m24.812584488s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:03.370378590 (14h16m25.114435452s) [upload=14695]: Done uploading segments... Aug 7 23:56:03.370401742 (14h16m25.114458656s) [upload=14695]: Upload finished. err: <nil> Aug 7 23:56:41.636734874 (14h17m3.380791742s) [upload=14707]: Starting upload Aug 7 23:56:41.636844337 (14h17m3.380901201s) [upload=14707]: Done starting upload Aug 7 23:56:41.636938361 (14h17m3.380995223s) [upload=14707]: Uploading segments... Aug 7 23:56:41.637004566 (14h17m3.381061436s) [upload=14707]: Got next segment. Inline: true Aug 7 23:56:41.637011720 (14h17m3.381068581s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:56:41.637073380 (14h17m3.381130242s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:41.637079684 (14h17m3.381136546s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:41.637083892 (14h17m3.381140753s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:41.369074470 (14h18m3.113131318s) [upload=14707]: Done uploading segments... Aug 7 23:57:41.369096313 (14h18m3.11315316s) [upload=14707]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:41.785857243 (14h18m3.529914104s) [upload=14726]: Starting upload Aug 7 23:57:41.785973295 (14h18m3.530030155s) [upload=14726]: Done starting upload Aug 7 23:57:41.786053418 (14h18m3.530110268s) [upload=14726]: Uploading segments... Aug 7 23:57:41.786115407 (14h18m3.530172273s) [upload=14726]: Got next segment. Inline: true Aug 7 23:57:41.786121268 (14h18m3.530178127s) [upload=14726]: Waiting for error group managing segments... Aug 7 23:57:41.786162041 (14h18m3.530218901s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:41.786167247 (14h18m3.530224108s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:41.786170583 (14h18m3.530227443s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.518215592 (14h19m3.262272452s) [upload=14726]: Done uploading segments... Aug 7 23:58:41.518237426 (14h19m3.262294275s) [upload=14726]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:49.080161447 (14h19m10.824218292s) [upload=14741]: Starting upload Aug 7 23:58:49.080236560 (14h19m10.824293407s) [upload=14741]: Done starting upload Aug 7 23:58:49.080313897 (14h19m10.824370744s) [upload=14741]: Uploading segments... Aug 7 23:58:49.080371825 (14h19m10.824428691s) [upload=14741]: Got next segment. Inline: true Aug 7 23:58:49.080376725 (14h19m10.824433586s) [upload=14741]: Waiting for error group managing segments... Aug 7 23:58:49.080418334 (14h19m10.824475197s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:49.080422560 (14h19m10.824479419s) [upload=14741]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:49.080438323 (14h19m10.824495186s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:49.220444289 (14h19m10.964501139s) [upload=14741]: Done uploading segments... Aug 7 23:58:49.220462660 (14h19m10.96451951s) [upload=14741]: Upload finished. err: <nil> Aug 7 23:59:26.271206280 (14h19m48.015263132s) [upload=14752]: Starting upload Aug 7 23:59:26.271300842 (14h19m48.015357706s) [upload=14752]: Done starting upload Aug 7 23:59:26.271409542 (14h19m48.015466389s) [upload=14752]: Uploading segments... Aug 7 23:59:26.271471969 (14h19m48.015528839s) [upload=14752]: Got next segment. Inline: true Aug 7 23:59:26.271478422 (14h19m48.015535286s) [upload=14752]: Waiting for error group managing segments... Aug 7 23:59:26.271524304 (14h19m48.015581168s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.271531803 (14h19m48.01558867s) [upload=14752]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:26.271535740 (14h19m48.015592604s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.006104454 (14h20m47.750161323s) [upload=14752]: Done uploading segments... Aug 8 00:00:26.006140966 (14h20m47.750197816s) [upload=14752]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:26.408809053 (14h20m48.152865927s) [upload=14766]: Starting upload Aug 8 00:00:26.408886561 (14h20m48.152943434s) [upload=14766]: Done starting upload Aug 8 00:00:26.408975679 (14h20m48.15303253s) [upload=14766]: Uploading segments... Aug 8 00:00:26.409028539 (14h20m48.153085476s) [upload=14766]: Got next segment. Inline: true Aug 8 00:00:26.409034850 (14h20m48.153091712s) [upload=14766]: Waiting for error group managing segments... Aug 8 00:00:26.409085190 (14h20m48.153142042s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:26.409090930 (14h20m48.153147793s) [upload=14766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:26.409094742 (14h20m48.153151605s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.549853950 (14h20m48.2939108s) [upload=14766]: Done uploading segments... Aug 8 00:00:26.549878419 (14h20m48.293935265s) [upload=14766]: Upload finished. err: <nil> Aug 8 00:01:08.274172938 (14h21m30.018229799s) [upload=14780]: Starting upload Aug 8 00:01:08.274248443 (14h21m30.018305302s) [upload=14780]: Done starting upload Aug 8 00:01:08.274326739 (14h21m30.018383585s) [upload=14780]: Uploading segments... Aug 8 00:01:08.274380689 (14h21m30.018437549s) [upload=14780]: Got next segment. Inline: true Aug 8 00:01:08.274385704 (14h21m30.018442565s) [upload=14780]: Waiting for error group managing segments... Aug 8 00:01:08.274437335 (14h21m30.018494182s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:08.274442821 (14h21m30.018499667s) [upload=14780]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:08.274445609 (14h21m30.018502454s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:08.005343335 (14h22m29.749400197s) [upload=14780]: Done uploading segments... Aug 8 00:02:08.005365121 (14h22m29.749421982s) [upload=14780]: Upload finished. err: metaclient: context deadline exceeded Duration: 977.758512ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [..........X...XX.X...XX.....................................] duration: 906.460433ms FAIL (2025-08-08 00:28:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 140355 (14h13m13.036197215s) [upload=14641]: Starting upload Aug 7 23:52:51.292216133 (14h13m13.036272985s) [upload=14641]: Done starting upload Aug 7 23:52:51.292307446 (14h13m13.036364307s) [upload=14641]: Uploading segments... Aug 7 23:52:51.292389087 (14h13m13.036445956s) [upload=14641]: Got next segment. Inline: true Aug 7 23:52:51.292394933 (14h13m13.036451793s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:52:51.292456032 (14h13m13.036512883s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:51.292461787 (14h13m13.036518647s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:51.292467676 (14h13m13.036524537s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:51.436915828 (14h13m13.180972678s) [upload=14641]: Done uploading segments... Aug 7 23:52:51.436934458 (14h13m13.18099131s) [upload=14641]: Upload finished. err: <nil> Aug 7 23:53:10.946261153 (14h13m32.690318028s) [upload=14646]: Starting upload Aug 7 23:53:10.946350198 (14h13m32.690407065s) [upload=14646]: Done starting upload Aug 7 23:53:10.946435423 (14h13m32.690492276s) [upload=14646]: Uploading segments... Aug 7 23:53:10.946489255 (14h13m32.690546124s) [upload=14646]: Got next segment. Inline: true Aug 7 23:53:10.946497660 (14h13m32.690554519s) [upload=14646]: Waiting for error group managing segments... Aug 7 23:53:10.946542678 (14h13m32.690599529s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:10.946548267 (14h13m32.690605128s) [upload=14646]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:10.946553792 (14h13m32.690610652s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:14.554651436 (14h13m36.298708309s) [upload=14646]: Done uploading segments... Aug 7 23:53:14.554675183 (14h13m36.298732045s) [upload=14646]: Upload finished. err: <nil> Aug 7 23:53:53.584502029 (14h14m15.328558897s) [upload=14658]: Starting upload Aug 7 23:53:53.584599757 (14h14m15.328656616s) [upload=14658]: Done starting upload Aug 7 23:53:53.584702364 (14h14m15.328759218s) [upload=14658]: Uploading segments... Aug 7 23:53:53.584798610 (14h14m15.32885548s) [upload=14658]: Got next segment. Inline: true Aug 7 23:53:53.584806594 (14h14m15.328863461s) [upload=14658]: Waiting for error group managing segments... Aug 7 23:53:53.584875193 (14h14m15.328932062s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:53.584882324 (14h14m15.328939193s) [upload=14658]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:53.584901657 (14h14m15.328958528s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:23.318821682 (14h14m45.062878542s) [upload=14658]: Done uploading segments... Aug 7 23:54:23.318863033 (14h14m45.062919884s) [upload=14658]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:36.775209359 (14h14m58.519266209s) [upload=14671]: Starting upload Aug 7 23:54:36.775292235 (14h14m58.519349112s) [upload=14671]: Done starting upload Aug 7 23:54:36.775412031 (14h14m58.519468883s) [upload=14671]: Uploading segments... Aug 7 23:54:36.775480767 (14h14m58.519537636s) [upload=14671]: Got next segment. Inline: true Aug 7 23:54:36.775487320 (14h14m58.519544181s) [upload=14671]: Waiting for error group managing segments... Aug 7 23:54:36.775539752 (14h14m58.519596612s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:36.775545893 (14h14m58.519602753s) [upload=14671]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:36.775549619 (14h14m58.519606479s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:36.926276427 (14h14m58.670333277s) [upload=14671]: Done uploading segments... Aug 7 23:54:36.926295206 (14h14m58.670352057s) [upload=14671]: Upload finished. err: <nil> Aug 7 23:54:57.675347843 (14h15m19.419404694s) [upload=14676]: Starting upload Aug 7 23:54:57.675412352 (14h15m19.419469212s) [upload=14676]: Done starting upload Aug 7 23:54:57.675475670 (14h15m19.41953252s) [upload=14676]: Uploading segments... Aug 7 23:54:57.675516336 (14h15m19.419573205s) [upload=14676]: Got next segment. Inline: true Aug 7 23:54:57.675521071 (14h15m19.419577931s) [upload=14676]: Waiting for error group managing segments... Aug 7 23:54:57.675565512 (14h15m19.419622375s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:57.675574255 (14h15m19.419631202s) [upload=14676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:57.675580694 (14h15m19.419637563s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:27.407552485 (14h15m49.151609344s) [upload=14676]: Done uploading segments... Aug 7 23:55:27.407592978 (14h15m49.151649829s) [upload=14676]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:06.851646232 (14h16m28.595703082s) [upload=14696]: Starting upload Aug 7 23:56:06.851717330 (14h16m28.595774191s) [upload=14696]: Done starting upload Aug 7 23:56:06.851786008 (14h16m28.59584286s) [upload=14696]: Uploading segments... Aug 7 23:56:06.851834232 (14h16m28.595891101s) [upload=14696]: Got next segment. Inline: true Aug 7 23:56:06.851839675 (14h16m28.595896535s) [upload=14696]: Waiting for error group managing segments... Aug 7 23:56:06.851896275 (14h16m28.595953125s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:06.851901659 (14h16m28.59595852s) [upload=14696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:06.851905505 (14h16m28.595962364s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:07.006202008 (14h16m28.750258868s) [upload=14696]: Done uploading segments... Aug 7 23:56:07.006222000 (14h16m28.750278861s) [upload=14696]: Upload finished. err: <nil> Aug 7 23:56:59.020763574 (14h17m20.764820421s) [upload=14712]: Starting upload Aug 7 23:56:59.020858955 (14h17m20.764915806s) [upload=14712]: Done starting upload Aug 7 23:56:59.020935444 (14h17m20.764992295s) [upload=14712]: Uploading segments... Aug 7 23:56:59.021007501 (14h17m20.76506437s) [upload=14712]: Got next segment. Inline: true Aug 7 23:56:59.021013631 (14h17m20.765070491s) [upload=14712]: Waiting for error group managing segments... Aug 7 23:56:59.021070471 (14h17m20.76512733s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:59.021076966 (14h17m20.765133821s) [upload=14712]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:59.021081305 (14h17m20.76513816s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:59.154820708 (14h17m20.898877569s) [upload=14712]: Done uploading segments... Aug 7 23:56:59.154839835 (14h17m20.898896685s) [upload=14712]: Upload finished. err: <nil> Aug 7 23:58:10.963677657 (14h18m32.707734508s) [upload=14728]: Starting upload Aug 7 23:58:10.963770332 (14h18m32.70782718s) [upload=14728]: Done starting upload Aug 7 23:58:10.963851725 (14h18m32.707908586s) [upload=14728]: Uploading segments... Aug 7 23:58:10.963901559 (14h18m32.707958431s) [upload=14728]: Got next segment. Inline: true Aug 7 23:58:10.963907663 (14h18m32.707964525s) [upload=14728]: Waiting for error group managing segments... Aug 7 23:58:10.963960167 (14h18m32.708017032s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:10.963966034 (14h18m32.708022896s) [upload=14728]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:10.963969944 (14h18m32.708026806s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.106069715 (14h18m32.850126571s) [upload=14728]: Done uploading segments... Aug 7 23:58:11.106112099 (14h18m32.850168961s) [upload=14728]: Upload finished. err: <nil> Aug 7 23:59:19.078128070 (14h19m40.822184931s) [upload=14749]: Starting upload Aug 7 23:59:19.078217499 (14h19m40.822274359s) [upload=14749]: Done starting upload Aug 7 23:59:19.078291040 (14h19m40.822347895s) [upload=14749]: Uploading segments... Aug 7 23:59:19.078333863 (14h19m40.822390732s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:19.078339134 (14h19m40.822395984s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:19.078391868 (14h19m40.822448717s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:19.078397213 (14h19m40.822454064s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:19.078401054 (14h19m40.822457905s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:48.808547234 (14h20m10.552604081s) [upload=14749]: Done uploading segments... Aug 7 23:59:48.808568563 (14h20m10.552625409s) [upload=14749]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:20.896255037 (14h20m42.640311905s) [upload=14765]: Starting upload Aug 8 00:00:20.896381648 (14h20m42.640438517s) [upload=14765]: Done starting upload Aug 8 00:00:20.896532416 (14h20m42.640589282s) [upload=14765]: Uploading segments... Aug 8 00:00:20.896610121 (14h20m42.640666994s) [upload=14765]: Got next segment. Inline: true Aug 8 00:00:20.896618785 (14h20m42.640675649s) [upload=14765]: Waiting for error group managing segments... Aug 8 00:00:20.896682486 (14h20m42.640739601s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:20.896690306 (14h20m42.640747187s) [upload=14765]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:20.896694946 (14h20m42.640751814s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:50.607106781 (14h21m12.351163663s) [upload=14765]: Done uploading segments... Aug 8 00:00:50.607134684 (14h21m12.351191558s) [upload=14765]: Upload finished. err: metaclient: context deadline exceeded Duration: 906.494008ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X.XX...XX.X..X.................................] duration: 2.444811023s FAIL (2025-08-08 00:28:15 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.444844323s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [........X.XXX.X.X.....X.....................................] duration: 967.097058ms FAIL (2025-08-08 00:28:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 967.135538ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [.............XX..XX.X.X.....................................] duration: 857.837576ms FAIL (2025-08-08 00:27:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0]: Starting upload Aug 7 23:53:17.153036870 (14h13m38.89709373s) [upload=14650]: Done starting upload Aug 7 23:53:17.153154678 (14h13m38.897211544s) [upload=14650]: Uploading segments... Aug 7 23:53:17.153233089 (14h13m38.897289957s) [upload=14650]: Got next segment. Inline: true Aug 7 23:53:17.153240256 (14h13m38.897297124s) [upload=14650]: Waiting for error group managing segments... Aug 7 23:53:17.153318892 (14h13m38.897375757s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:17.153325816 (14h13m38.89738268s) [upload=14650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:17.153330453 (14h13m38.897387314s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:16.875757982 (14h14m38.619814845s) [upload=14650]: Done uploading segments... Aug 7 23:54:16.875781849 (14h14m38.619838701s) [upload=14650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:24.158448119 (14h14m45.90250498s) [upload=14667]: Starting upload Aug 7 23:54:24.158525946 (14h14m45.902582806s) [upload=14667]: Done starting upload Aug 7 23:54:24.158598972 (14h14m45.902655824s) [upload=14667]: Uploading segments... Aug 7 23:54:24.158654148 (14h14m45.902711017s) [upload=14667]: Got next segment. Inline: true Aug 7 23:54:24.158659995 (14h14m45.902716856s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:54:24.158714896 (14h14m45.902771747s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:24.158720357 (14h14m45.902777209s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:24.158724105 (14h14m45.902780965s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:23.888400190 (14h15m45.63245704s) [upload=14667]: Done uploading segments... Aug 7 23:55:23.888419087 (14h15m45.632475938s) [upload=14667]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:24.297146309 (14h15m46.041203174s) [upload=14687]: Starting upload Aug 7 23:55:24.297232144 (14h15m46.04128901s) [upload=14687]: Done starting upload Aug 7 23:55:24.297315772 (14h15m46.041372619s) [upload=14687]: Uploading segments... Aug 7 23:55:24.297396238 (14h15m46.041453151s) [upload=14687]: Got next segment. Inline: true Aug 7 23:55:24.297403729 (14h15m46.04146059s) [upload=14687]: Waiting for error group managing segments... Aug 7 23:55:24.297444116 (14h15m46.041500976s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:24.297450971 (14h15m46.041507831s) [upload=14687]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:24.297453883 (14h15m46.041510749s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:24.439126204 (14h15m46.183183055s) [upload=14687]: Done uploading segments... Aug 7 23:55:24.439141121 (14h15m46.183197972s) [upload=14687]: Upload finished. err: <nil> Aug 7 23:56:03.068174127 (14h16m24.812230977s) [upload=14695]: Starting upload Aug 7 23:56:03.068258755 (14h16m24.812315617s) [upload=14695]: Done starting upload Aug 7 23:56:03.068368961 (14h16m24.812425812s) [upload=14695]: Uploading segments... Aug 7 23:56:03.068437851 (14h16m24.812494719s) [upload=14695]: Got next segment. Inline: true Aug 7 23:56:03.068445058 (14h16m24.812501927s) [upload=14695]: Waiting for error group managing segments... Aug 7 23:56:03.068516175 (14h16m24.812573049s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:03.068523275 (14h16m24.812580137s) [upload=14695]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:03.068527513 (14h16m24.812584488s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:03.370378590 (14h16m25.114435452s) [upload=14695]: Done uploading segments... Aug 7 23:56:03.370401742 (14h16m25.114458656s) [upload=14695]: Upload finished. err: <nil> Aug 7 23:56:41.636734874 (14h17m3.380791742s) [upload=14707]: Starting upload Aug 7 23:56:41.636844337 (14h17m3.380901201s) [upload=14707]: Done starting upload Aug 7 23:56:41.636938361 (14h17m3.380995223s) [upload=14707]: Uploading segments... Aug 7 23:56:41.637004566 (14h17m3.381061436s) [upload=14707]: Got next segment. Inline: true Aug 7 23:56:41.637011720 (14h17m3.381068581s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:56:41.637073380 (14h17m3.381130242s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:41.637079684 (14h17m3.381136546s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:41.637083892 (14h17m3.381140753s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:41.369074470 (14h18m3.113131318s) [upload=14707]: Done uploading segments... Aug 7 23:57:41.369096313 (14h18m3.11315316s) [upload=14707]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:41.785857243 (14h18m3.529914104s) [upload=14726]: Starting upload Aug 7 23:57:41.785973295 (14h18m3.530030155s) [upload=14726]: Done starting upload Aug 7 23:57:41.786053418 (14h18m3.530110268s) [upload=14726]: Uploading segments... Aug 7 23:57:41.786115407 (14h18m3.530172273s) [upload=14726]: Got next segment. Inline: true Aug 7 23:57:41.786121268 (14h18m3.530178127s) [upload=14726]: Waiting for error group managing segments... Aug 7 23:57:41.786162041 (14h18m3.530218901s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:41.786167247 (14h18m3.530224108s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:41.786170583 (14h18m3.530227443s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.518215592 (14h19m3.262272452s) [upload=14726]: Done uploading segments... Aug 7 23:58:41.518237426 (14h19m3.262294275s) [upload=14726]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:49.080161447 (14h19m10.824218292s) [upload=14741]: Starting upload Aug 7 23:58:49.080236560 (14h19m10.824293407s) [upload=14741]: Done starting upload Aug 7 23:58:49.080313897 (14h19m10.824370744s) [upload=14741]: Uploading segments... Aug 7 23:58:49.080371825 (14h19m10.824428691s) [upload=14741]: Got next segment. Inline: true Aug 7 23:58:49.080376725 (14h19m10.824433586s) [upload=14741]: Waiting for error group managing segments... Aug 7 23:58:49.080418334 (14h19m10.824475197s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:49.080422560 (14h19m10.824479419s) [upload=14741]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:49.080438323 (14h19m10.824495186s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:49.220444289 (14h19m10.964501139s) [upload=14741]: Done uploading segments... Aug 7 23:58:49.220462660 (14h19m10.96451951s) [upload=14741]: Upload finished. err: <nil> Aug 7 23:59:26.271206280 (14h19m48.015263132s) [upload=14752]: Starting upload Aug 7 23:59:26.271300842 (14h19m48.015357706s) [upload=14752]: Done starting upload Aug 7 23:59:26.271409542 (14h19m48.015466389s) [upload=14752]: Uploading segments... Aug 7 23:59:26.271471969 (14h19m48.015528839s) [upload=14752]: Got next segment. Inline: true Aug 7 23:59:26.271478422 (14h19m48.015535286s) [upload=14752]: Waiting for error group managing segments... Aug 7 23:59:26.271524304 (14h19m48.015581168s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.271531803 (14h19m48.01558867s) [upload=14752]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:26.271535740 (14h19m48.015592604s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.006104454 (14h20m47.750161323s) [upload=14752]: Done uploading segments... Aug 8 00:00:26.006140966 (14h20m47.750197816s) [upload=14752]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:26.408809053 (14h20m48.152865927s) [upload=14766]: Starting upload Aug 8 00:00:26.408886561 (14h20m48.152943434s) [upload=14766]: Done starting upload Aug 8 00:00:26.408975679 (14h20m48.15303253s) [upload=14766]: Uploading segments... Aug 8 00:00:26.409028539 (14h20m48.153085476s) [upload=14766]: Got next segment. Inline: true Aug 8 00:00:26.409034850 (14h20m48.153091712s) [upload=14766]: Waiting for error group managing segments... Aug 8 00:00:26.409085190 (14h20m48.153142042s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:26.409090930 (14h20m48.153147793s) [upload=14766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:26.409094742 (14h20m48.153151605s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.549853950 (14h20m48.2939108s) [upload=14766]: Done uploading segments... Aug 8 00:00:26.549878419 (14h20m48.293935265s) [upload=14766]: Upload finished. err: <nil> Aug 8 00:01:08.274172938 (14h21m30.018229799s) [upload=14780]: Starting upload Aug 8 00:01:08.274248443 (14h21m30.018305302s) [upload=14780]: Done starting upload Aug 8 00:01:08.274326739 (14h21m30.018383585s) [upload=14780]: Uploading segments... Aug 8 00:01:08.274380689 (14h21m30.018437549s) [upload=14780]: Got next segment. Inline: true Aug 8 00:01:08.274385704 (14h21m30.018442565s) [upload=14780]: Waiting for error group managing segments... Aug 8 00:01:08.274437335 (14h21m30.018494182s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:08.274442821 (14h21m30.018499667s) [upload=14780]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:08.274445609 (14h21m30.018502454s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:08.005343335 (14h22m29.749400197s) [upload=14780]: Done uploading segments... Aug 8 00:02:08.005365121 (14h22m29.749421982s) [upload=14780]: Upload finished. err: metaclient: context deadline exceeded Duration: 857.873512ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 1.065560092s FAIL (2025-08-08 00:27:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.065682711s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............XX.X.X...XX...................................] duration: 2.522184019s FAIL (2025-08-08 00:27:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 6]: Uploading piece... Aug 7 23:57:10.893816491 (14h17m32.637873354s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.894285524 (14h17m32.638342391s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895348932 (14h17m32.639405787s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895932061 (14h17m32.639988921s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.896545320 (14h17m32.640602184s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897117472 (14h17m32.641174341s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897621325 (14h17m32.641678192s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898101338 (14h17m32.642158204s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898693607 (14h17m32.642750474s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.900512475 (14h17m32.644569335s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.902337804 (14h17m32.646394669s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.904215667 (14h17m32.648272537s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.906034481 (14h17m32.650091365s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.907857426 (14h17m32.651914295s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.908666188 (14h17m32.652723069s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:11.510238622 (14h17m33.254295484s) [upload=14716]: Next returned nil segment Aug 7 23:57:11.510261464 (14h17m33.254318324s) [upload=14716]: Waiting for error group managing segments... Aug 7 23:57:11.657094388 (14h17m33.401151236s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.674408374 (14h17m33.418465234s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.681203586 (14h17m33.425260449s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.682780850 (14h17m33.4268377s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.684042315 (14h17m33.428099176s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.719658547 (14h17m33.463715401s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.730348391 (14h17m33.474405242s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.748264399 (14h17m33.492321249s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.769764028 (14h17m33.513820878s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.774177874 (14h17m33.518234733s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.789864472 (14h17m33.533921326s) [noise=true , address=45.91.116.13:4153 , piece_id=WBFFQT5I6ZR5ZKUJRFULNOJZ455KIJMA5EEATDDOCVFUKRD67NFA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.799519597 (14h17m33.543576447s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800753821 (14h17m33.544810667s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800785983 (14h17m33.544842842s) [seg_pos={0 0}, upload=14716]: Segment reached optimal threshold of 13 pieces. Aug 7 23:57:11.801026052 (14h17m33.545082898s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801098020 (14h17m33.545154867s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801174944 (14h17m33.54523179s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection Aug 7 23:57:11.801184926 (14h17m33.545241773s) [seg_pos={0 0}, upload=14716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:57:11.801257096 (14h17m33.545313956s) [upload=14716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:11.801261009 (14h17m33.545317869s) [upload=14716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:11.942784613 (14h17m33.686841513s) [upload=14716]: Done uploading segments... Aug 7 23:57:11.942815424 (14h17m33.686872279s) [upload=14716]: Upload finished. err: <nil> Aug 7 23:58:20.561350310 (14h18m42.305407184s) [upload=14732]: Starting upload Aug 7 23:58:20.561483402 (14h18m42.30554027s) [upload=14732]: Done starting upload Aug 7 23:58:20.561650297 (14h18m42.305707159s) [upload=14732]: Uploading segments... Aug 7 23:58:20.562196341 (14h18m42.306253212s) [upload=14732]: Got next segment. Inline: false Aug 7 23:58:20.562334166 (14h18m42.306391031s) [upload=14732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:20.562343255 (14h18m42.306400125s) [upload=14732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:50.281195862 (14h19m12.025252712s) [upload=14732]: Next segment err: context deadline exceeded Aug 7 23:58:50.281383705 (14h19m12.025440555s) [upload=14732]: Done uploading segments... Aug 7 23:58:50.281405600 (14h19m12.02546245s) [upload=14732]: Upload finished. err: context deadline exceeded Aug 7 23:58:50.716777496 (14h19m12.460834346s) [upload=14742]: Starting upload Aug 7 23:58:50.716857905 (14h19m12.460914751s) [upload=14742]: Done starting upload Aug 7 23:58:50.716999972 (14h19m12.461056825s) [upload=14742]: Uploading segments... Aug 7 23:58:50.717186124 (14h19m12.461242976s) [upload=14742]: Got next segment. Inline: false Aug 7 23:58:50.717245601 (14h19m12.461302466s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:50.717252185 (14h19m12.461309053s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:20.436529435 (14h19m42.180586305s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:20.436734882 (14h19m42.180791733s) [upload=14742]: Done uploading segments... Aug 7 23:59:20.436756200 (14h19m42.180813054s) [upload=14742]: Upload finished. err: context deadline exceeded Duration: 2.522214812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................XXX..XX....................................] duration: 2.652161164s FAIL (2025-08-08 00:27:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 807513521 (14h17m32.551570367s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:55.062716703 (14h18m16.806773554s) [seg_pos={0 0}, upload=14717]: Begin upload segment... Aug 7 23:57:55.062772247 (14h18m16.806829106s) [seg_pos={0 0}, upload=14717]: Done begin upload segment. Aug 7 23:57:55.062817614 (14h18m16.806874477s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.062833275 (14h18m16.806890146s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063321348 (14h18m16.807378211s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063625241 (14h18m16.8076821s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.064591461 (14h18m16.808648334s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065249737 (14h18m16.809306608s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065844042 (14h18m16.809900909s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.066738904 (14h18m16.810795771s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067295868 (14h18m16.81135274s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067868039 (14h18m16.811924903s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068378475 (14h18m16.812435346s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068951556 (14h18m16.81300842s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069355853 (14h18m16.813412717s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069707384 (14h18m16.813764245s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070103904 (14h18m16.814160763s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070434322 (14h18m16.814491182s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070775335 (14h18m16.814832199s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.072531247 (14h18m16.816588107s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.642344995 (14h18m17.386401842s) [upload=14717]: Next returned nil segment Aug 7 23:57:55.642365727 (14h18m17.38642259s) [upload=14717]: Waiting for error group managing segments... Aug 7 23:57:55.820297514 (14h18m17.564354366s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.832273438 (14h18m17.576330291s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.845342869 (14h18m17.58939972s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.850001106 (14h18m17.594057955s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874147776 (14h18m17.618204625s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874555951 (14h18m17.6186128s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.890084811 (14h18m17.634141672s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.901998021 (14h18m17.646054868s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.930296357 (14h18m17.674353212s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.008745690 (14h18m17.752802541s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.041830348 (14h18m17.785887195s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.086405192 (14h18m17.830462041s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.092724415 (14h18m17.83678127s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095750120 (14h18m17.839806985s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095784368 (14h18m17.839841231s) [seg_pos={0 0}, upload=14717]: Segment reached optimal threshold of 14 pieces. Aug 7 23:57:56.095960472 (14h18m17.840017334s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096160491 (14h18m17.840217355s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096190692 (14h18m17.840247554s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eu7SDhEXZGS6HeAXFZWfxoci2oPdjtYvZ5Bbhvae2neKTxYKo): protocol: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection Aug 7 23:57:56.096286416 (14h18m17.840343264s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w4v4QV7PxhXTwuxbvTQST2buoWNySrziRYTo9Mb8dpENd9XCX): protocol: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection Aug 7 23:57:56.096300170 (14h18m17.840357019s) [seg_pos={0 0}, upload=14717]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:57:56.096419305 (14h18m17.840476164s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:56.096426206 (14h18m17.84048307s) [upload=14717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:30.094636522 (14h22m51.838693376s) [upload=14717]: Done uploading segments... Aug 8 00:02:30.094667105 (14h22m51.838723955s) [upload=14717]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.652197412s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.906812657s FAIL (2025-08-08 00:27:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.906852631s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 178.89771ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-08 00:27:40 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 178.929617ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [...........X...XX.X...XX....................................] duration: 1.041582091s FAIL (2025-08-08 00:27:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 140355 (14h13m13.036197215s) [upload=14641]: Starting upload Aug 7 23:52:51.292216133 (14h13m13.036272985s) [upload=14641]: Done starting upload Aug 7 23:52:51.292307446 (14h13m13.036364307s) [upload=14641]: Uploading segments... Aug 7 23:52:51.292389087 (14h13m13.036445956s) [upload=14641]: Got next segment. Inline: true Aug 7 23:52:51.292394933 (14h13m13.036451793s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:52:51.292456032 (14h13m13.036512883s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:51.292461787 (14h13m13.036518647s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:51.292467676 (14h13m13.036524537s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:51.436915828 (14h13m13.180972678s) [upload=14641]: Done uploading segments... Aug 7 23:52:51.436934458 (14h13m13.18099131s) [upload=14641]: Upload finished. err: <nil> Aug 7 23:53:10.946261153 (14h13m32.690318028s) [upload=14646]: Starting upload Aug 7 23:53:10.946350198 (14h13m32.690407065s) [upload=14646]: Done starting upload Aug 7 23:53:10.946435423 (14h13m32.690492276s) [upload=14646]: Uploading segments... Aug 7 23:53:10.946489255 (14h13m32.690546124s) [upload=14646]: Got next segment. Inline: true Aug 7 23:53:10.946497660 (14h13m32.690554519s) [upload=14646]: Waiting for error group managing segments... Aug 7 23:53:10.946542678 (14h13m32.690599529s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:10.946548267 (14h13m32.690605128s) [upload=14646]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:10.946553792 (14h13m32.690610652s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:14.554651436 (14h13m36.298708309s) [upload=14646]: Done uploading segments... Aug 7 23:53:14.554675183 (14h13m36.298732045s) [upload=14646]: Upload finished. err: <nil> Aug 7 23:53:53.584502029 (14h14m15.328558897s) [upload=14658]: Starting upload Aug 7 23:53:53.584599757 (14h14m15.328656616s) [upload=14658]: Done starting upload Aug 7 23:53:53.584702364 (14h14m15.328759218s) [upload=14658]: Uploading segments... Aug 7 23:53:53.584798610 (14h14m15.32885548s) [upload=14658]: Got next segment. Inline: true Aug 7 23:53:53.584806594 (14h14m15.328863461s) [upload=14658]: Waiting for error group managing segments... Aug 7 23:53:53.584875193 (14h14m15.328932062s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:53.584882324 (14h14m15.328939193s) [upload=14658]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:53.584901657 (14h14m15.328958528s) [upload=14658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:23.318821682 (14h14m45.062878542s) [upload=14658]: Done uploading segments... Aug 7 23:54:23.318863033 (14h14m45.062919884s) [upload=14658]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:36.775209359 (14h14m58.519266209s) [upload=14671]: Starting upload Aug 7 23:54:36.775292235 (14h14m58.519349112s) [upload=14671]: Done starting upload Aug 7 23:54:36.775412031 (14h14m58.519468883s) [upload=14671]: Uploading segments... Aug 7 23:54:36.775480767 (14h14m58.519537636s) [upload=14671]: Got next segment. Inline: true Aug 7 23:54:36.775487320 (14h14m58.519544181s) [upload=14671]: Waiting for error group managing segments... Aug 7 23:54:36.775539752 (14h14m58.519596612s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:36.775545893 (14h14m58.519602753s) [upload=14671]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:36.775549619 (14h14m58.519606479s) [upload=14671]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:36.926276427 (14h14m58.670333277s) [upload=14671]: Done uploading segments... Aug 7 23:54:36.926295206 (14h14m58.670352057s) [upload=14671]: Upload finished. err: <nil> Aug 7 23:54:57.675347843 (14h15m19.419404694s) [upload=14676]: Starting upload Aug 7 23:54:57.675412352 (14h15m19.419469212s) [upload=14676]: Done starting upload Aug 7 23:54:57.675475670 (14h15m19.41953252s) [upload=14676]: Uploading segments... Aug 7 23:54:57.675516336 (14h15m19.419573205s) [upload=14676]: Got next segment. Inline: true Aug 7 23:54:57.675521071 (14h15m19.419577931s) [upload=14676]: Waiting for error group managing segments... Aug 7 23:54:57.675565512 (14h15m19.419622375s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:57.675574255 (14h15m19.419631202s) [upload=14676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:57.675580694 (14h15m19.419637563s) [upload=14676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:27.407552485 (14h15m49.151609344s) [upload=14676]: Done uploading segments... Aug 7 23:55:27.407592978 (14h15m49.151649829s) [upload=14676]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:06.851646232 (14h16m28.595703082s) [upload=14696]: Starting upload Aug 7 23:56:06.851717330 (14h16m28.595774191s) [upload=14696]: Done starting upload Aug 7 23:56:06.851786008 (14h16m28.59584286s) [upload=14696]: Uploading segments... Aug 7 23:56:06.851834232 (14h16m28.595891101s) [upload=14696]: Got next segment. Inline: true Aug 7 23:56:06.851839675 (14h16m28.595896535s) [upload=14696]: Waiting for error group managing segments... Aug 7 23:56:06.851896275 (14h16m28.595953125s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:06.851901659 (14h16m28.59595852s) [upload=14696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:06.851905505 (14h16m28.595962364s) [upload=14696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:07.006202008 (14h16m28.750258868s) [upload=14696]: Done uploading segments... Aug 7 23:56:07.006222000 (14h16m28.750278861s) [upload=14696]: Upload finished. err: <nil> Aug 7 23:56:59.020763574 (14h17m20.764820421s) [upload=14712]: Starting upload Aug 7 23:56:59.020858955 (14h17m20.764915806s) [upload=14712]: Done starting upload Aug 7 23:56:59.020935444 (14h17m20.764992295s) [upload=14712]: Uploading segments... Aug 7 23:56:59.021007501 (14h17m20.76506437s) [upload=14712]: Got next segment. Inline: true Aug 7 23:56:59.021013631 (14h17m20.765070491s) [upload=14712]: Waiting for error group managing segments... Aug 7 23:56:59.021070471 (14h17m20.76512733s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:59.021076966 (14h17m20.765133821s) [upload=14712]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:59.021081305 (14h17m20.76513816s) [upload=14712]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:59.154820708 (14h17m20.898877569s) [upload=14712]: Done uploading segments... Aug 7 23:56:59.154839835 (14h17m20.898896685s) [upload=14712]: Upload finished. err: <nil> Aug 7 23:58:10.963677657 (14h18m32.707734508s) [upload=14728]: Starting upload Aug 7 23:58:10.963770332 (14h18m32.70782718s) [upload=14728]: Done starting upload Aug 7 23:58:10.963851725 (14h18m32.707908586s) [upload=14728]: Uploading segments... Aug 7 23:58:10.963901559 (14h18m32.707958431s) [upload=14728]: Got next segment. Inline: true Aug 7 23:58:10.963907663 (14h18m32.707964525s) [upload=14728]: Waiting for error group managing segments... Aug 7 23:58:10.963960167 (14h18m32.708017032s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:10.963966034 (14h18m32.708022896s) [upload=14728]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:10.963969944 (14h18m32.708026806s) [upload=14728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.106069715 (14h18m32.850126571s) [upload=14728]: Done uploading segments... Aug 7 23:58:11.106112099 (14h18m32.850168961s) [upload=14728]: Upload finished. err: <nil> Aug 7 23:59:19.078128070 (14h19m40.822184931s) [upload=14749]: Starting upload Aug 7 23:59:19.078217499 (14h19m40.822274359s) [upload=14749]: Done starting upload Aug 7 23:59:19.078291040 (14h19m40.822347895s) [upload=14749]: Uploading segments... Aug 7 23:59:19.078333863 (14h19m40.822390732s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:19.078339134 (14h19m40.822395984s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:19.078391868 (14h19m40.822448717s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:19.078397213 (14h19m40.822454064s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:19.078401054 (14h19m40.822457905s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:48.808547234 (14h20m10.552604081s) [upload=14749]: Done uploading segments... Aug 7 23:59:48.808568563 (14h20m10.552625409s) [upload=14749]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:20.896255037 (14h20m42.640311905s) [upload=14765]: Starting upload Aug 8 00:00:20.896381648 (14h20m42.640438517s) [upload=14765]: Done starting upload Aug 8 00:00:20.896532416 (14h20m42.640589282s) [upload=14765]: Uploading segments... Aug 8 00:00:20.896610121 (14h20m42.640666994s) [upload=14765]: Got next segment. Inline: true Aug 8 00:00:20.896618785 (14h20m42.640675649s) [upload=14765]: Waiting for error group managing segments... Aug 8 00:00:20.896682486 (14h20m42.640739601s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:20.896690306 (14h20m42.640747187s) [upload=14765]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:20.896694946 (14h20m42.640751814s) [upload=14765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:50.607106781 (14h21m12.351163663s) [upload=14765]: Done uploading segments... Aug 8 00:00:50.607134684 (14h21m12.351191558s) [upload=14765]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.041617091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..............X.................] duration: 3.208317225s FAIL (2025-08-08 00:27:34 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== oad Aug 8 00:12:40.688890451 (14h33m2.432947302s) [upload=14976]: Uploading segments... Aug 8 00:12:40.689315108 (14h33m2.433371968s) [upload=14976]: Got next segment. Inline: false Aug 8 00:12:40.689365721 (14h33m2.433422571s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:12:40.689371637 (14h33m2.433428498s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:12:40.938712718 (14h33m2.682769583s) [seg_pos={0 0}, upload=14976]: Begin upload segment... Aug 8 00:12:40.938780540 (14h33m2.68283739s) [seg_pos={0 0}, upload=14976]: Done begin upload segment. Aug 8 00:12:40.938826511 (14h33m2.682883372s) [noise=true , address=104.36.82.11:4122 , piece_id=T4ATL4PZUGFLXWE47UGHDXX3ID2LEJKQRGQQMF2G4J42TET2JVYQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.938837427 (14h33m2.682894292s) [noise=true , address=64.49.13.169:4715 , piece_id=TKX4G5GWLDNDPR66UADVNNWV2TAFGCVEMWZAU4GROJXFX3Q537TA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939397428 (14h33m2.683454296s) [noise=true , address=104.36.82.44:4881 , piece_id=TS5LURERTPPJMH3KOK3ABNKEWSWOF7IUNNKP5DONPQTXKIF4S55Q, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939417193 (14h33m2.683474059s) [noise=true , address=104.36.82.5:4859 , piece_id=FHBE3H7NTIY3PHCICRNYDGY7MEILS27N4O5DLKQQWM5UXELX3EWQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939726153 (14h33m2.683783022s) [noise=true , address=104.36.82.45:4937 , piece_id=4Z6T6K7EP3LS3XYSTOARORNVPGKX6HYIHV33I665O5FI4OPGOXNA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.939786076 (14h33m2.683842943s) [noise=true , address=104.36.82.23:4453 , piece_id=V7D5MY7E63SEALXMVDJOS26NYQDQAFJ4SHUISCK6TPKDQKKI4Z3Q, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940057147 (14h33m2.68411401s) [noise=true , address=104.36.82.22:4537 , piece_id=3Q4MNZBN5UPYVL4BC673XQVPMLW3MYTY2CNLKFE7NW6ZWSFHAWPQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940208252 (14h33m2.684265105s) [noise=true , address=104.36.82.59:4313 , piece_id=LIWITE4IJVNHLO2DFFDI2M4D67UAHLM223EF6FAXGNGWIIMGH7FA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940408469 (14h33m2.68446533s) [noise=true , address=64.49.13.154:4930 , piece_id=OGQJV2HMKOE6IPET6A6JRENBWMGJ6NWSGL7Q55653IB6QBAUARQA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.940742041 (14h33m2.684798902s) [noise=true , address=45.91.116.7:4034 , piece_id=XJQV5V55F3KLAF5MZZMWZN7HE6TRHQ2V6BDDMKOWMDMKEBAP6UZQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941021709 (14h33m2.685078579s) [noise=true , address=104.36.82.37:4671 , piece_id=HYA6VNL24EERU27SJLYO7TCV7DQIIH7Q4LXAEA5BXSHBTIXCHKDQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941299121 (14h33m2.685355986s) [noise=true , address=104.36.82.43:4855 , piece_id=AMNGI7UV2PCEQ2U4LAOXIFT4T3WXPD7YHLR77VR7MSSQGHESY5EA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941588088 (14h33m2.685644951s) [noise=true , address=104.36.82.19:4401 , piece_id=NACQ6PBCDIHNJAQIFQUVYAPU2HY5C4ZDW6QMJBGY3E45TNWHFQNQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941637056 (14h33m2.685693925s) [noise=true , address=104.36.82.61:4404 , piece_id=F5YUVKGDN4MPHJKOLR5DRVABOU6326LGWAJXKL4RCWZI35TLMXKQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.941950914 (14h33m2.686007788s) [noise=true , address=79.127.201.215:4351 , piece_id=DR37C77MS5YTW7DK7XOJQGMEXWOSUKOD76SZY7TXLIB5RLXLEVMQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.942345807 (14h33m2.686402678s) [noise=true , address=104.36.82.55:4206 , piece_id=FEFQ3F2EFHMZI53KUIAGHESBATXL5S2EOAARLOHJ2YDSMGVLZRQQ, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.942560802 (14h33m2.686617667s) [noise=true , address=104.36.82.12:4158 , piece_id=SCBLDCSKC6HTOUIUA43BA4452TPHEIVKCMOSVHR42YXVUWYUFPRA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:40.943202713 (14h33m2.687259576s) [noise=true , address=193.149.164.143:4693 , piece_id=P5YP7XDPKE2VYGTIJMAD6TV2LAPXJRN47E3PDPDDLVNI73VJGUBA, seg_pos={0 0}, upload=14976]: Uploading piece... Aug 8 00:12:41.525441834 (14h33m3.269498698s) [upload=14976]: Next returned nil segment Aug 8 00:12:41.525462664 (14h33m3.269519531s) [upload=14976]: Waiting for error group managing segments... Aug 8 00:12:41.699976519 (14h33m3.444033384s) [noise=true , address=104.36.82.19:4401 , piece_id=NACQ6PBCDIHNJAQIFQUVYAPU2HY5C4ZDW6QMJBGY3E45TNWHFQNQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.700944219 (14h33m3.445001137s) [noise=true , address=104.36.82.23:4453 , piece_id=V7D5MY7E63SEALXMVDJOS26NYQDQAFJ4SHUISCK6TPKDQKKI4Z3Q, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.702379298 (14h33m3.446436158s) [noise=true , address=104.36.82.37:4671 , piece_id=HYA6VNL24EERU27SJLYO7TCV7DQIIH7Q4LXAEA5BXSHBTIXCHKDQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.710467855 (14h33m3.454524716s) [noise=true , address=104.36.82.55:4206 , piece_id=FEFQ3F2EFHMZI53KUIAGHESBATXL5S2EOAARLOHJ2YDSMGVLZRQQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.716048399 (14h33m3.46010525s) [noise=true , address=104.36.82.22:4537 , piece_id=3Q4MNZBN5UPYVL4BC673XQVPMLW3MYTY2CNLKFE7NW6ZWSFHAWPQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.769601981 (14h33m3.513658832s) [noise=true , address=104.36.82.44:4881 , piece_id=TS5LURERTPPJMH3KOK3ABNKEWSWOF7IUNNKP5DONPQTXKIF4S55Q, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.774083597 (14h33m3.518140448s) [noise=true , address=104.36.82.5:4859 , piece_id=FHBE3H7NTIY3PHCICRNYDGY7MEILS27N4O5DLKQQWM5UXELX3EWQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.790244473 (14h33m3.534301324s) [noise=true , address=45.91.116.7:4034 , piece_id=XJQV5V55F3KLAF5MZZMWZN7HE6TRHQ2V6BDDMKOWMDMKEBAP6UZQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.790597987 (14h33m3.534654837s) [noise=true , address=104.36.82.61:4404 , piece_id=F5YUVKGDN4MPHJKOLR5DRVABOU6326LGWAJXKL4RCWZI35TLMXKQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.814650455 (14h33m3.558707307s) [noise=true , address=104.36.82.59:4313 , piece_id=LIWITE4IJVNHLO2DFFDI2M4D67UAHLM223EF6FAXGNGWIIMGH7FA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.916929838 (14h33m3.660986699s) [noise=true , address=104.36.82.12:4158 , piece_id=SCBLDCSKC6HTOUIUA43BA4452TPHEIVKCMOSVHR42YXVUWYUFPRA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:41.946542933 (14h33m3.690599791s) [noise=true , address=79.127.201.215:4351 , piece_id=DR37C77MS5YTW7DK7XOJQGMEXWOSUKOD76SZY7TXLIB5RLXLEVMQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.014532160 (14h33m3.75858901s) [noise=true , address=104.36.82.43:4855 , piece_id=AMNGI7UV2PCEQ2U4LAOXIFT4T3WXPD7YHLR77VR7MSSQGHESY5EA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.157107757 (14h33m3.901164607s) [noise=true , address=104.36.82.11:4122 , piece_id=T4ATL4PZUGFLXWE47UGHDXX3ID2LEJKQRGQQMF2G4J42TET2JVYQ, seg_pos={0 0}, upload=14976]: Done uploading piece. err: <nil> Aug 8 00:12:42.157146638 (14h33m3.901203502s) [seg_pos={0 0}, upload=14976]: Segment reached optimal threshold of 14 pieces. Aug 8 00:12:42.157376876 (14h33m3.901433737s) [noise=true , address=64.49.13.169:4715 , piece_id=TKX4G5GWLDNDPR66UADVNNWV2TAFGCVEMWZAU4GROJXFX3Q537TA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157574941 (14h33m3.901631791s) [noise=true , address=193.149.164.143:4693 , piece_id=P5YP7XDPKE2VYGTIJMAD6TV2LAPXJRN47E3PDPDDLVNI73VJGUBA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LHoCE9CnM32tbMNqAB7HpmArSVLTycv9o6kSSFNUJsoDeERkQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157631621 (14h33m3.901688486s) [noise=true , address=64.49.13.154:4930 , piece_id=OGQJV2HMKOE6IPET6A6JRENBWMGJ6NWSGL7Q55653IB6QBAUARQA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kNK8y5et7MiiWNDT6iaC8YBAk3PX6WX1XSuU4QMSFBp3KymBb): protocol: expected piece hash; context canceled; EOF Aug 8 00:12:42.157680271 (14h33m3.901737121s) [noise=true , address=104.36.82.45:4937 , piece_id=4Z6T6K7EP3LS3XYSTOARORNVPGKX6HYIHV33I665O5FI4OPGOXNA, seg_pos={0 0}, upload=14976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection; write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51352->104.36.82.45:4937: use of closed network connection Aug 8 00:12:42.157698358 (14h33m3.901755219s) [seg_pos={0 0}, upload=14976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:12:42.157816415 (14h33m3.901873289s) [upload=14976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:12:42.157839617 (14h33m3.901896479s) [upload=14976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:12:42.510566001 (14h33m4.254622851s) [upload=14976]: Done uploading segments... Aug 8 00:12:42.510584893 (14h33m4.254641743s) [upload=14976]: Upload finished. err: <nil> Duration: 3.208354771s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................XXX.X.X.....................................] duration: 3.307411511s FAIL (2025-08-08 00:27:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.307445117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............XX.X.X...XX..................................] duration: 2.421532306s FAIL (2025-08-08 00:27:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 6]: Uploading piece... Aug 7 23:57:10.893816491 (14h17m32.637873354s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.894285524 (14h17m32.638342391s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895348932 (14h17m32.639405787s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.895932061 (14h17m32.639988921s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.896545320 (14h17m32.640602184s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897117472 (14h17m32.641174341s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.897621325 (14h17m32.641678192s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898101338 (14h17m32.642158204s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.898693607 (14h17m32.642750474s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.900512475 (14h17m32.644569335s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.902337804 (14h17m32.646394669s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.904215667 (14h17m32.648272537s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.906034481 (14h17m32.650091365s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.907857426 (14h17m32.651914295s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:10.908666188 (14h17m32.652723069s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Uploading piece... Aug 7 23:57:11.510238622 (14h17m33.254295484s) [upload=14716]: Next returned nil segment Aug 7 23:57:11.510261464 (14h17m33.254318324s) [upload=14716]: Waiting for error group managing segments... Aug 7 23:57:11.657094388 (14h17m33.401151236s) [noise=true , address=77.90.31.10:4518 , piece_id=LU6UTDDO6HUIGXV2QXCL45CIBFQMOQGS7CTQHBRKCBE2JJKLI6FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.674408374 (14h17m33.418465234s) [noise=true , address=77.90.31.15:4698 , piece_id=WBU562MADMX3KSCRCD2H5R7MQN5IPFBLKYSKU47WMGYYBKZGPYJA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.681203586 (14h17m33.425260449s) [noise=true , address=77.90.31.12:4589 , piece_id=5I5BMAQARX67CB2RRMCXG2U7KELJ6HSHRT3573EH3ZX6NJ257R4A, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.682780850 (14h17m33.4268377s) [noise=true , address=77.90.31.21:4877 , piece_id=SF7XDMNH7KGS26UE6MKUK3VLFZ6IOP6WZKMVO4H6SIAULM623GLA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.684042315 (14h17m33.428099176s) [noise=true , address=104.36.82.16:4294 , piece_id=SIMVCA63FAX5EQZJO5TBOSGPVYLFOAICTABKMXS5SCQGNOUPFBIA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.719658547 (14h17m33.463715401s) [noise=true , address=45.91.116.12:4225 , piece_id=NTUY2VEJFWSFOAGWUPTJYNYNWL3XMK4C3FJUE6MSTMTS2TZJ7VUA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.730348391 (14h17m33.474405242s) [noise=true , address=77.90.31.21:4878 , piece_id=XAXEYDOX54GUKMI22UKTA6BSTSY5WYATTITTGSRQ3BBYDGDCO5DQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.748264399 (14h17m33.492321249s) [noise=true , address=77.90.31.19:4841 , piece_id=Q3XFTIVGOSRQARSFONAKX26VAFTJF7THPIQV6USKI7N2LN3JBTLQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.769764028 (14h17m33.513820878s) [noise=true , address=45.91.116.6:4975 , piece_id=VDEP57UE6G4CBU7DOYMMOIRXJUTTWPJKDJDGCGUOHBGVPAYCM6PA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.774177874 (14h17m33.518234733s) [noise=true , address=77.90.31.14:4661 , piece_id=WFDXKM726LBL4MH4CJ4AQIUSXTQCQ3BJV34QBUGWPHNFFPZYGZKA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.789864472 (14h17m33.533921326s) [noise=true , address=45.91.116.13:4153 , piece_id=WBFFQT5I6ZR5ZKUJRFULNOJZ455KIJMA5EEATDDOCVFUKRD67NFA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.799519597 (14h17m33.543576447s) [noise=true , address=45.91.116.12:4227 , piece_id=J2FM3FAWY5HYXX3VFC6CSHUSWDQDNKUUB6NL6MWSF2ZG3BZFWBEA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800753821 (14h17m33.544810667s) [noise=true , address=77.90.31.13:4626 , piece_id=PR6GY2Y7HDUORSLYZXU4IVLVA6RIMBFSRFJW4OTLXUZM7MJZ4KOQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: <nil> Aug 7 23:57:11.800785983 (14h17m33.544842842s) [seg_pos={0 0}, upload=14716]: Segment reached optimal threshold of 13 pieces. Aug 7 23:57:11.801026052 (14h17m33.545082898s) [noise=true , address=104.36.82.14:4222 , piece_id=JFWGS6VCDTFOQYG32ZFTWNYLFGVVQPXD7K6TZXC4FR2SGEOUS7FA, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801098020 (14h17m33.545154867s) [noise=true , address=104.36.82.10:4078 , piece_id=3DPLCXZLQWMZSYSSO3JPLZFP2VXJ5NKYO4FDVFYOWJLT43C7XSTQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:11.801174944 (14h17m33.54523179s) [noise=true , address=104.36.82.5:4839 , piece_id=AV3ZBQ4W44IZEDAPHJGHKK5D5VRNU5XGOQGDWEARNGIKWRQAGYBQ, seg_pos={0 0}, upload=14716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48474->104.36.82.5:4839: use of closed network connection Aug 7 23:57:11.801184926 (14h17m33.545241773s) [seg_pos={0 0}, upload=14716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:57:11.801257096 (14h17m33.545313956s) [upload=14716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:11.801261009 (14h17m33.545317869s) [upload=14716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:11.942784613 (14h17m33.686841513s) [upload=14716]: Done uploading segments... Aug 7 23:57:11.942815424 (14h17m33.686872279s) [upload=14716]: Upload finished. err: <nil> Aug 7 23:58:20.561350310 (14h18m42.305407184s) [upload=14732]: Starting upload Aug 7 23:58:20.561483402 (14h18m42.30554027s) [upload=14732]: Done starting upload Aug 7 23:58:20.561650297 (14h18m42.305707159s) [upload=14732]: Uploading segments... Aug 7 23:58:20.562196341 (14h18m42.306253212s) [upload=14732]: Got next segment. Inline: false Aug 7 23:58:20.562334166 (14h18m42.306391031s) [upload=14732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:20.562343255 (14h18m42.306400125s) [upload=14732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:50.281195862 (14h19m12.025252712s) [upload=14732]: Next segment err: context deadline exceeded Aug 7 23:58:50.281383705 (14h19m12.025440555s) [upload=14732]: Done uploading segments... Aug 7 23:58:50.281405600 (14h19m12.02546245s) [upload=14732]: Upload finished. err: context deadline exceeded Aug 7 23:58:50.716777496 (14h19m12.460834346s) [upload=14742]: Starting upload Aug 7 23:58:50.716857905 (14h19m12.460914751s) [upload=14742]: Done starting upload Aug 7 23:58:50.716999972 (14h19m12.461056825s) [upload=14742]: Uploading segments... Aug 7 23:58:50.717186124 (14h19m12.461242976s) [upload=14742]: Got next segment. Inline: false Aug 7 23:58:50.717245601 (14h19m12.461302466s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:50.717252185 (14h19m12.461309053s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:20.436529435 (14h19m42.180586305s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:20.436734882 (14h19m42.180791733s) [upload=14742]: Done uploading segments... Aug 7 23:59:20.436756200 (14h19m42.180813054s) [upload=14742]: Upload finished. err: context deadline exceeded Duration: 2.421567997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................XXX..XX...................................] duration: 2.787091384s FAIL (2025-08-08 00:27:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 807513521 (14h17m32.551570367s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:55.062716703 (14h18m16.806773554s) [seg_pos={0 0}, upload=14717]: Begin upload segment... Aug 7 23:57:55.062772247 (14h18m16.806829106s) [seg_pos={0 0}, upload=14717]: Done begin upload segment. Aug 7 23:57:55.062817614 (14h18m16.806874477s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.062833275 (14h18m16.806890146s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063321348 (14h18m16.807378211s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.063625241 (14h18m16.8076821s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.064591461 (14h18m16.808648334s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065249737 (14h18m16.809306608s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.065844042 (14h18m16.809900909s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.066738904 (14h18m16.810795771s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067295868 (14h18m16.81135274s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.067868039 (14h18m16.811924903s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068378475 (14h18m16.812435346s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.068951556 (14h18m16.81300842s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069355853 (14h18m16.813412717s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.069707384 (14h18m16.813764245s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070103904 (14h18m16.814160763s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070434322 (14h18m16.814491182s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.070775335 (14h18m16.814832199s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.072531247 (14h18m16.816588107s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Uploading piece... Aug 7 23:57:55.642344995 (14h18m17.386401842s) [upload=14717]: Next returned nil segment Aug 7 23:57:55.642365727 (14h18m17.38642259s) [upload=14717]: Waiting for error group managing segments... Aug 7 23:57:55.820297514 (14h18m17.564354366s) [noise=true , address=104.36.82.8:4996 , piece_id=YC2AWTI2UUHJ3TAETROO7NOZUZGZ6C72B3IN576XF4NBKCBMKLFQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.832273438 (14h18m17.576330291s) [noise=true , address=104.36.82.22:4539 , piece_id=J3W3WDIR4K2AE6A7GOA3XSCQMQIXTKX4QOW3A3CNYLPMGLPMX3OQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.845342869 (14h18m17.58939972s) [noise=true , address=104.36.82.58:4281 , piece_id=DMQFQHOJLY5JI7Z6FY4YKV35QLCAYBJM5KRAXPQBUFSDMYXQUZ4A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.850001106 (14h18m17.594057955s) [noise=true , address=104.36.82.12:4175 , piece_id=ZNZ3LXHM74KKEPVMCHCCAWTGTHA7OL2AQ4OABCZ7WM2TURDVM6QA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874147776 (14h18m17.618204625s) [noise=true , address=104.36.82.23:4477 , piece_id=LT2OZ725COADSVAQACMRKHQXNONJE6HYK6CPAPYTWBYAIBOZDM5A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.874555951 (14h18m17.6186128s) [noise=true , address=45.91.116.7:4023 , piece_id=KPAEZGGSBFF5CWHLUNR2MQSYU462G3HAFRKZT43DFZUDP5HSL57A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.890084811 (14h18m17.634141672s) [noise=true , address=104.36.82.45:4928 , piece_id=SIQVBTS33NGZBV2USAHXOZM27GIKGBZPZL3US6WURCCMGI72JVEQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.901998021 (14h18m17.646054868s) [noise=true , address=104.36.82.14:4251 , piece_id=P6OSQG6KNMPC5LCYTOIEJ7G6V2A6CM2ATGUI32OSYKBRBV5KJBGA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:55.930296357 (14h18m17.674353212s) [noise=true , address=45.91.116.6:4982 , piece_id=WJM5DM6B3UV4A574XGJAFFEWQUNBDTTNLCC25BO4L63WLPJRRHBA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.008745690 (14h18m17.752802541s) [noise=true , address=104.36.82.43:4866 , piece_id=7KRB44Z2OLODJFZC6YUGVZUTTMJZ5RMNTYQTTKWFIP7EVKA5LJGQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.041830348 (14h18m17.785887195s) [noise=true , address=104.36.82.35:4613 , piece_id=YBDNA55YNS6QJGH2INDXJKSIEM522P3OMEGQUXGGTNHSRXZFOVMQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.086405192 (14h18m17.830462041s) [noise=true , address=104.36.82.2:4737 , piece_id=GBZDLCAMK54PLKK6U2B4ICGKTCNYYOKIIAOHVIS4Z6BQSVH6WKHA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.092724415 (14h18m17.83678127s) [noise=true , address=104.36.82.9:4021 , piece_id=2DEM4H4BS5XIBGOKNITL65YWPA6H6422YVIOGUFTO34YWS4YNG7A, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095750120 (14h18m17.839806985s) [noise=true , address=45.91.116.8:4051 , piece_id=EWR3NMINSYDDDULIWXFMKTXBYR6VN2V4BNK2IKTFYRFZ253ZKFOQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: <nil> Aug 7 23:57:56.095784368 (14h18m17.839841231s) [seg_pos={0 0}, upload=14717]: Segment reached optimal threshold of 14 pieces. Aug 7 23:57:56.095960472 (14h18m17.840017334s) [noise=true , address=64.49.13.155:4914 , piece_id=V33ONJ4B3FNLTUBOAXNBUG5UVYYPYAZZM3A6EAZBDXTFO7OGJTFA, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096160491 (14h18m17.840217355s) [noise=true , address=64.49.13.155:4923 , piece_id=EFJVLJHSGTYX54TTFD4O55GF342ZTVNEGQCFY3HQMEWM36QEXN6Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF Aug 7 23:57:56.096190692 (14h18m17.840247554s) [noise=true , address=104.36.82.59:4330 , piece_id=73BPLWXHEWDSB5YDFQG7TXJLDVBSE755ENCHHM6APOYXWGRBBO7Q, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eu7SDhEXZGS6HeAXFZWfxoci2oPdjtYvZ5Bbhvae2neKTxYKo): protocol: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54270->104.36.82.59:4330: use of closed network connection Aug 7 23:57:56.096286416 (14h18m17.840343264s) [noise=true , address=104.36.82.37:4668 , piece_id=JUUEHTRRQCBKOM36HHV7B3SIF57KL326TH2KJU57CIRATVZHKFBQ, seg_pos={0 0}, upload=14717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w4v4QV7PxhXTwuxbvTQST2buoWNySrziRYTo9Mb8dpENd9XCX): protocol: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47936->104.36.82.37:4668: use of closed network connection Aug 7 23:57:56.096300170 (14h18m17.840357019s) [seg_pos={0 0}, upload=14717]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:57:56.096419305 (14h18m17.840476164s) [upload=14717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:57:56.096426206 (14h18m17.84048307s) [upload=14717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:30.094636522 (14h22m51.838693376s) [upload=14717]: Done uploading segments... Aug 8 00:02:30.094667105 (14h22m51.838723955s) [upload=14717]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.787126393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...X................................] duration: 1.12213771s FAIL (2025-08-08 00:27:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.122164962s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [..............XX..XX.X.X....................................] duration: 881.661081ms FAIL (2025-08-08 00:27:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0]: Starting upload Aug 7 23:53:17.153036870 (14h13m38.89709373s) [upload=14650]: Done starting upload Aug 7 23:53:17.153154678 (14h13m38.897211544s) [upload=14650]: Uploading segments... Aug 7 23:53:17.153233089 (14h13m38.897289957s) [upload=14650]: Got next segment. Inline: true Aug 7 23:53:17.153240256 (14h13m38.897297124s) [upload=14650]: Waiting for error group managing segments... Aug 7 23:53:17.153318892 (14h13m38.897375757s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:17.153325816 (14h13m38.89738268s) [upload=14650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:17.153330453 (14h13m38.897387314s) [upload=14650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:16.875757982 (14h14m38.619814845s) [upload=14650]: Done uploading segments... Aug 7 23:54:16.875781849 (14h14m38.619838701s) [upload=14650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:24.158448119 (14h14m45.90250498s) [upload=14667]: Starting upload Aug 7 23:54:24.158525946 (14h14m45.902582806s) [upload=14667]: Done starting upload Aug 7 23:54:24.158598972 (14h14m45.902655824s) [upload=14667]: Uploading segments... Aug 7 23:54:24.158654148 (14h14m45.902711017s) [upload=14667]: Got next segment. Inline: true Aug 7 23:54:24.158659995 (14h14m45.902716856s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:54:24.158714896 (14h14m45.902771747s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:24.158720357 (14h14m45.902777209s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:24.158724105 (14h14m45.902780965s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:23.888400190 (14h15m45.63245704s) [upload=14667]: Done uploading segments... Aug 7 23:55:23.888419087 (14h15m45.632475938s) [upload=14667]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:24.297146309 (14h15m46.041203174s) [upload=14687]: Starting upload Aug 7 23:55:24.297232144 (14h15m46.04128901s) [upload=14687]: Done starting upload Aug 7 23:55:24.297315772 (14h15m46.041372619s) [upload=14687]: Uploading segments... Aug 7 23:55:24.297396238 (14h15m46.041453151s) [upload=14687]: Got next segment. Inline: true Aug 7 23:55:24.297403729 (14h15m46.04146059s) [upload=14687]: Waiting for error group managing segments... Aug 7 23:55:24.297444116 (14h15m46.041500976s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:24.297450971 (14h15m46.041507831s) [upload=14687]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:24.297453883 (14h15m46.041510749s) [upload=14687]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:24.439126204 (14h15m46.183183055s) [upload=14687]: Done uploading segments... Aug 7 23:55:24.439141121 (14h15m46.183197972s) [upload=14687]: Upload finished. err: <nil> Aug 7 23:56:03.068174127 (14h16m24.812230977s) [upload=14695]: Starting upload Aug 7 23:56:03.068258755 (14h16m24.812315617s) [upload=14695]: Done starting upload Aug 7 23:56:03.068368961 (14h16m24.812425812s) [upload=14695]: Uploading segments... Aug 7 23:56:03.068437851 (14h16m24.812494719s) [upload=14695]: Got next segment. Inline: true Aug 7 23:56:03.068445058 (14h16m24.812501927s) [upload=14695]: Waiting for error group managing segments... Aug 7 23:56:03.068516175 (14h16m24.812573049s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:03.068523275 (14h16m24.812580137s) [upload=14695]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:03.068527513 (14h16m24.812584488s) [upload=14695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:03.370378590 (14h16m25.114435452s) [upload=14695]: Done uploading segments... Aug 7 23:56:03.370401742 (14h16m25.114458656s) [upload=14695]: Upload finished. err: <nil> Aug 7 23:56:41.636734874 (14h17m3.380791742s) [upload=14707]: Starting upload Aug 7 23:56:41.636844337 (14h17m3.380901201s) [upload=14707]: Done starting upload Aug 7 23:56:41.636938361 (14h17m3.380995223s) [upload=14707]: Uploading segments... Aug 7 23:56:41.637004566 (14h17m3.381061436s) [upload=14707]: Got next segment. Inline: true Aug 7 23:56:41.637011720 (14h17m3.381068581s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:56:41.637073380 (14h17m3.381130242s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:41.637079684 (14h17m3.381136546s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:41.637083892 (14h17m3.381140753s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:41.369074470 (14h18m3.113131318s) [upload=14707]: Done uploading segments... Aug 7 23:57:41.369096313 (14h18m3.11315316s) [upload=14707]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:41.785857243 (14h18m3.529914104s) [upload=14726]: Starting upload Aug 7 23:57:41.785973295 (14h18m3.530030155s) [upload=14726]: Done starting upload Aug 7 23:57:41.786053418 (14h18m3.530110268s) [upload=14726]: Uploading segments... Aug 7 23:57:41.786115407 (14h18m3.530172273s) [upload=14726]: Got next segment. Inline: true Aug 7 23:57:41.786121268 (14h18m3.530178127s) [upload=14726]: Waiting for error group managing segments... Aug 7 23:57:41.786162041 (14h18m3.530218901s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:41.786167247 (14h18m3.530224108s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:41.786170583 (14h18m3.530227443s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.518215592 (14h19m3.262272452s) [upload=14726]: Done uploading segments... Aug 7 23:58:41.518237426 (14h19m3.262294275s) [upload=14726]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:49.080161447 (14h19m10.824218292s) [upload=14741]: Starting upload Aug 7 23:58:49.080236560 (14h19m10.824293407s) [upload=14741]: Done starting upload Aug 7 23:58:49.080313897 (14h19m10.824370744s) [upload=14741]: Uploading segments... Aug 7 23:58:49.080371825 (14h19m10.824428691s) [upload=14741]: Got next segment. Inline: true Aug 7 23:58:49.080376725 (14h19m10.824433586s) [upload=14741]: Waiting for error group managing segments... Aug 7 23:58:49.080418334 (14h19m10.824475197s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:49.080422560 (14h19m10.824479419s) [upload=14741]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:49.080438323 (14h19m10.824495186s) [upload=14741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:49.220444289 (14h19m10.964501139s) [upload=14741]: Done uploading segments... Aug 7 23:58:49.220462660 (14h19m10.96451951s) [upload=14741]: Upload finished. err: <nil> Aug 7 23:59:26.271206280 (14h19m48.015263132s) [upload=14752]: Starting upload Aug 7 23:59:26.271300842 (14h19m48.015357706s) [upload=14752]: Done starting upload Aug 7 23:59:26.271409542 (14h19m48.015466389s) [upload=14752]: Uploading segments... Aug 7 23:59:26.271471969 (14h19m48.015528839s) [upload=14752]: Got next segment. Inline: true Aug 7 23:59:26.271478422 (14h19m48.015535286s) [upload=14752]: Waiting for error group managing segments... Aug 7 23:59:26.271524304 (14h19m48.015581168s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.271531803 (14h19m48.01558867s) [upload=14752]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:26.271535740 (14h19m48.015592604s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.006104454 (14h20m47.750161323s) [upload=14752]: Done uploading segments... Aug 8 00:00:26.006140966 (14h20m47.750197816s) [upload=14752]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:26.408809053 (14h20m48.152865927s) [upload=14766]: Starting upload Aug 8 00:00:26.408886561 (14h20m48.152943434s) [upload=14766]: Done starting upload Aug 8 00:00:26.408975679 (14h20m48.15303253s) [upload=14766]: Uploading segments... Aug 8 00:00:26.409028539 (14h20m48.153085476s) [upload=14766]: Got next segment. Inline: true Aug 8 00:00:26.409034850 (14h20m48.153091712s) [upload=14766]: Waiting for error group managing segments... Aug 8 00:00:26.409085190 (14h20m48.153142042s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:26.409090930 (14h20m48.153147793s) [upload=14766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:26.409094742 (14h20m48.153151605s) [upload=14766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:26.549853950 (14h20m48.2939108s) [upload=14766]: Done uploading segments... Aug 8 00:00:26.549878419 (14h20m48.293935265s) [upload=14766]: Upload finished. err: <nil> Aug 8 00:01:08.274172938 (14h21m30.018229799s) [upload=14780]: Starting upload Aug 8 00:01:08.274248443 (14h21m30.018305302s) [upload=14780]: Done starting upload Aug 8 00:01:08.274326739 (14h21m30.018383585s) [upload=14780]: Uploading segments... Aug 8 00:01:08.274380689 (14h21m30.018437549s) [upload=14780]: Got next segment. Inline: true Aug 8 00:01:08.274385704 (14h21m30.018442565s) [upload=14780]: Waiting for error group managing segments... Aug 8 00:01:08.274437335 (14h21m30.018494182s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:08.274442821 (14h21m30.018499667s) [upload=14780]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:08.274445609 (14h21m30.018502454s) [upload=14780]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:08.005343335 (14h22m29.749400197s) [upload=14780]: Done uploading segments... Aug 8 00:02:08.005365121 (14h22m29.749421982s) [upload=14780]: Upload finished. err: metaclient: context deadline exceeded Duration: 881.704968ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.753848272s FAIL (2025-08-08 00:27:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.753866943s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)