Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................................XXXXXX.XX............] duration: 2.239826025s FAIL (2025-08-07 23:28:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.239862264s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [........................................XXX...XX.X..........] duration: 1.088984742s FAIL (2025-08-07 23:28:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.089007781s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 291.751059ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:28:40 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 291.778522ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X...X.XX...........] duration: 3.918934958s FAIL (2025-08-07 23:28:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== inished. err: context deadline exceeded Aug 7 23:20:07.202369543 (13h40m28.946426394s) [upload=14082]: Starting upload Aug 7 23:20:07.202426657 (13h40m28.946483509s) [upload=14082]: Done starting upload Aug 7 23:20:07.202696950 (13h40m28.946753814s) [upload=14082]: Uploading segments... Aug 7 23:20:07.202761256 (13h40m28.94681811s) [upload=14082]: Got next segment. Inline: false Aug 7 23:20:07.202807046 (13h40m28.946863908s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:07.202814233 (13h40m28.946871102s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:20:45.722093118 (13h41m7.466149983s) [seg_pos={0 0}, upload=14082]: Begin upload segment... Aug 7 23:20:45.722177165 (13h41m7.466234018s) [seg_pos={0 0}, upload=14082]: Done begin upload segment. Aug 7 23:20:45.722218538 (13h41m7.4662754s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722246298 (13h41m7.46630316s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722330454 (13h41m7.466387316s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722658583 (13h41m7.466715506s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722717191 (13h41m7.46677405s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722989701 (13h41m7.467046568s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723061064 (13h41m7.467117927s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723458393 (13h41m7.467515259s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.729867582 (13h41m7.473924444s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.731063643 (13h41m7.475120512s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.732350031 (13h41m7.476406894s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.733989697 (13h41m7.478046556s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734727605 (13h41m7.478784474s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734786637 (13h41m7.478843504s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735460668 (13h41m7.479517532s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735486537 (13h41m7.479543401s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.736565943 (13h41m7.480622804s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.737386328 (13h41m7.48144319s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:46.299198843 (13h41m8.043255689s) [upload=14082]: Next returned nil segment Aug 7 23:20:46.299217870 (13h41m8.043274733s) [upload=14082]: Waiting for error group managing segments... Aug 7 23:20:46.464881860 (13h41m8.208938721s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.473775294 (13h41m8.217832164s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.474664669 (13h41m8.218721541s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.479688336 (13h41m8.223745197s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.485171804 (13h41m8.229228658s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.486836180 (13h41m8.230893032s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.496476947 (13h41m8.240533796s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.556494206 (13h41m8.300551081s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.628743021 (13h41m8.372799882s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.645987246 (13h41m8.390044106s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.666511132 (13h41m8.410567992s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.671756371 (13h41m8.41581322s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.934025878 (13h41m8.678082729s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970907150 (13h41m8.714964016s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970940415 (13h41m8.714997268s) [seg_pos={0 0}, upload=14082]: Segment reached optimal threshold of 14 pieces. Aug 7 23:20:46.971273207 (13h41m8.715330074s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MD2UDNXEmeGra9fvhEb8eGL2R6vGLHtSKLpSyX3H3tZCe1r5B): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971460895 (13h41m8.715517755s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971541847 (13h41m8.715598706s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJTNVtPFN48S9ZrayehLD2HcVckyJWLx2rtEvyH56W22R8Suq): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971599101 (13h41m8.715655962s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JeDjqgszkkq2asUBCL5oTvH6X21TnWkkmqGEEzBbz1YctdDXX): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971618718 (13h41m8.715675568s) [seg_pos={0 0}, upload=14082]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:20:46.971698846 (13h41m8.715755698s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:20:46.971704848 (13h41m8.71576171s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:07.219673652 (13h41m28.963730505s) [upload=14082]: Done uploading segments... Aug 7 23:21:07.219701493 (13h41m28.963758344s) [upload=14082]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.918973693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [......................................XX..XXXXX.............] duration: 915.569163ms FAIL (2025-08-07 23:28:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:11:40.667902567 (13h32m2.411959413s) [upload=13933]: Starting upload Aug 7 23:11:40.667961940 (13h32m2.412018788s) [upload=13933]: Done starting upload Aug 7 23:11:40.668020524 (13h32m2.41207737s) [upload=13933]: Uploading segments... Aug 7 23:11:40.668075247 (13h32m2.412132113s) [upload=13933]: Got next segment. Inline: true Aug 7 23:11:40.668080664 (13h32m2.412137525s) [upload=13933]: Waiting for error group managing segments... Aug 7 23:11:40.668121190 (13h32m2.412178036s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:40.668125756 (13h32m2.412182616s) [upload=13933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:40.668128431 (13h32m2.412185291s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.394243254 (13h33m2.138300118s) [upload=13933]: Done uploading segments... Aug 7 23:12:40.394270270 (13h33m2.138327132s) [upload=13933]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:40.798630406 (13h33m2.542687267s) [upload=13953]: Starting upload Aug 7 23:12:40.798725809 (13h33m2.54278266s) [upload=13953]: Done starting upload Aug 7 23:12:40.798811946 (13h33m2.542868796s) [upload=13953]: Uploading segments... Aug 7 23:12:40.798867069 (13h33m2.542923932s) [upload=13953]: Got next segment. Inline: true Aug 7 23:12:40.798873108 (13h33m2.542929959s) [upload=13953]: Waiting for error group managing segments... Aug 7 23:12:40.798918573 (13h33m2.542975422s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:40.798924603 (13h33m2.542981463s) [upload=13953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:40.798939855 (13h33m2.542996716s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.942390421 (13h33m2.686447272s) [upload=13953]: Done uploading segments... Aug 7 23:12:40.942410212 (13h33m2.686467062s) [upload=13953]: Upload finished. err: <nil> Aug 7 23:13:18.137108974 (13h33m39.881165825s) [upload=13967]: Starting upload Aug 7 23:13:18.137180408 (13h33m39.881237292s) [upload=13967]: Done starting upload Aug 7 23:13:18.137258925 (13h33m39.881315786s) [upload=13967]: Uploading segments... Aug 7 23:13:18.137309635 (13h33m39.881366505s) [upload=13967]: Got next segment. Inline: true Aug 7 23:13:18.137314511 (13h33m39.881371371s) [upload=13967]: Waiting for error group managing segments... Aug 7 23:13:18.137374772 (13h33m39.881431633s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.137380412 (13h33m39.881437273s) [upload=13967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.137383733 (13h33m39.881440593s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.301480519 (13h33m40.045537379s) [upload=13967]: Done uploading segments... Aug 7 23:13:18.301498918 (13h33m40.045555767s) [upload=13967]: Upload finished. err: <nil> Aug 7 23:14:13.169029066 (13h34m34.913085913s) [upload=13980]: Starting upload Aug 7 23:14:13.169095254 (13h34m34.913152101s) [upload=13980]: Done starting upload Aug 7 23:14:13.169153338 (13h34m34.913210184s) [upload=13980]: Uploading segments... Aug 7 23:14:13.169225662 (13h34m34.913282508s) [upload=13980]: Got next segment. Inline: true Aug 7 23:14:13.169230885 (13h34m34.913287744s) [upload=13980]: Waiting for error group managing segments... Aug 7 23:14:13.169272521 (13h34m34.913329381s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:13.169288952 (13h34m34.913345813s) [upload=13980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:13.169291623 (13h34m34.913348483s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:12.895960606 (13h35m34.640017467s) [upload=13980]: Done uploading segments... Aug 7 23:15:12.895983608 (13h35m34.640040456s) [upload=13980]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:13.308146067 (13h35m35.052202918s) [upload=13997]: Starting upload Aug 7 23:15:13.308219271 (13h35m35.052276134s) [upload=13997]: Done starting upload Aug 7 23:15:13.308297319 (13h35m35.052354171s) [upload=13997]: Uploading segments... Aug 7 23:15:13.308360203 (13h35m35.052417072s) [upload=13997]: Got next segment. Inline: true Aug 7 23:15:13.308365548 (13h35m35.052422408s) [upload=13997]: Waiting for error group managing segments... Aug 7 23:15:13.308409794 (13h35m35.052466657s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:13.308426102 (13h35m35.052482963s) [upload=13997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:13.308429725 (13h35m35.052486585s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:13.040589970 (13h36m34.784646843s) [upload=13997]: Done uploading segments... Aug 7 23:16:13.040614428 (13h36m34.784671274s) [upload=13997]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:20.767866726 (13h36m42.511923577s) [upload=14018]: Starting upload Aug 7 23:16:20.767927544 (13h36m42.511984398s) [upload=14018]: Done starting upload Aug 7 23:16:20.767993090 (13h36m42.512049942s) [upload=14018]: Uploading segments... Aug 7 23:16:20.768055054 (13h36m42.512111926s) [upload=14018]: Got next segment. Inline: true Aug 7 23:16:20.768061500 (13h36m42.512118351s) [upload=14018]: Waiting for error group managing segments... Aug 7 23:16:20.768130604 (13h36m42.512187577s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:20.768139327 (13h36m42.512196192s) [upload=14018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:20.768144987 (13h36m42.512201849s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:20.496845379 (13h37m42.240902231s) [upload=14018]: Done uploading segments... Aug 7 23:17:20.496873778 (13h37m42.240930632s) [upload=14018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:20.916777280 (13h37m42.660834146s) [upload=14032]: Starting upload Aug 7 23:17:20.916877286 (13h37m42.660934149s) [upload=14032]: Done starting upload Aug 7 23:17:20.917002557 (13h37m42.661059408s) [upload=14032]: Uploading segments... Aug 7 23:17:20.917058206 (13h37m42.661115077s) [upload=14032]: Got next segment. Inline: true Aug 7 23:17:20.917064317 (13h37m42.661121178s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:17:20.917116347 (13h37m42.661173214s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:20.917123150 (13h37m42.661180014s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:20.917127779 (13h37m42.661184648s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:20.640884349 (13h38m42.384941214s) [upload=14032]: Done uploading segments... Aug 7 23:18:20.640914749 (13h38m42.384971617s) [upload=14032]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:21.050365938 (13h38m42.79442281s) [upload=14051]: Starting upload Aug 7 23:18:21.050484720 (13h38m42.794541587s) [upload=14051]: Done starting upload Aug 7 23:18:21.050589902 (13h38m42.794646752s) [upload=14051]: Uploading segments... Aug 7 23:18:21.050663797 (13h38m42.794720667s) [upload=14051]: Got next segment. Inline: true Aug 7 23:18:21.050670451 (13h38m42.794727311s) [upload=14051]: Waiting for error group managing segments... Aug 7 23:18:21.050719413 (13h38m42.794776275s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:21.050725163 (13h38m42.794782023s) [upload=14051]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:21.050728952 (13h38m42.794785811s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:20.780076504 (13h39m42.524133366s) [upload=14051]: Done uploading segments... Aug 7 23:19:20.780099469 (13h39m42.52415633s) [upload=14051]: Upload finished. err: metaclient: context deadline exceeded Duration: 915.60422ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.641424121s FAIL (2025-08-07 23:28:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.641455995s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X.XX..X...............] duration: 2.633630222s FAIL (2025-08-07 23:28:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.633656245s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................................XXXXXX.XX...........] duration: 2.309761816s FAIL (2025-08-07 23:28:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.309803993s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.........................................X.XXXXX.X..........] duration: 2.94506031s FAIL (2025-08-07 23:28:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.945088231s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X....X............X..............................] duration: 3.146155409s FAIL (2025-08-07 23:28:03 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 3.146187989s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXX...........] duration: 2.423982821s FAIL (2025-08-07 23:27:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== upload Aug 7 23:18:25.127322319 (13h38m46.871379183s) [upload=14052]: Done starting upload Aug 7 23:18:25.127477364 (13h38m46.871534223s) [upload=14052]: Uploading segments... Aug 7 23:18:25.127943505 (13h38m46.872000369s) [upload=14052]: Got next segment. Inline: false Aug 7 23:18:25.128022972 (13h38m46.87207984s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:25.128032328 (13h38m46.872089195s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:44.923639390 (13h39m6.667696251s) [seg_pos={0 0}, upload=14052]: Begin upload segment... Aug 7 23:18:44.923694548 (13h39m6.667751411s) [seg_pos={0 0}, upload=14052]: Done begin upload segment. Aug 7 23:18:44.923719227 (13h39m6.667776093s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.923742940 (13h39m6.667799806s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924062820 (13h39m6.66811968s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924231371 (13h39m6.668288235s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924310412 (13h39m6.668367275s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924552072 (13h39m6.668608933s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924795545 (13h39m6.66885246s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925037101 (13h39m6.669093961s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925293050 (13h39m6.669349911s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925526381 (13h39m6.669583247s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925876037 (13h39m6.669932897s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926278285 (13h39m6.670335152s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926413382 (13h39m6.670470243s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926732421 (13h39m6.670789284s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927153074 (13h39m6.671209983s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927375228 (13h39m6.671432108s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:45.586845311 (13h39m7.330902181s) [upload=14052]: Next returned nil segment Aug 7 23:18:45.586873064 (13h39m7.330929934s) [upload=14052]: Waiting for error group managing segments... Aug 7 23:18:45.727354327 (13h39m7.471411192s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.767876801 (13h39m7.511933648s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.792222311 (13h39m7.536279182s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.836090391 (13h39m7.580147242s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.847910967 (13h39m7.591967821s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.874060717 (13h39m7.618117569s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.882796683 (13h39m7.626853529s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.883366192 (13h39m7.62742305s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.899253981 (13h39m7.643310835s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.905112812 (13h39m7.64916966s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.911636868 (13h39m7.655693715s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.934683670 (13h39m7.67874052s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959619489 (13h39m7.703676335s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959647576 (13h39m7.703704436s) [seg_pos={0 0}, upload=14052]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:45.959785097 (13h39m7.703841946s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959873513 (13h39m7.703930376s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959997706 (13h39m7.704054568s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection Aug 7 23:18:45.960040133 (13h39m7.70409698s) [seg_pos={0 0}, upload=14052]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:45.960143346 (13h39m7.704200223s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:45.960151195 (13h39m7.704208074s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:34.514397794 (13h40m56.258454651s) [upload=14052]: Done uploading segments... Aug 7 23:20:34.514434819 (13h40m56.258491687s) [upload=14052]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:20:34.909564598 (13h40m56.65362148s) [upload=14088]: Starting upload Aug 7 23:20:34.909657672 (13h40m56.653714552s) [upload=14088]: Done starting upload Aug 7 23:20:34.909836458 (13h40m56.653893325s) [upload=14088]: Uploading segments... Aug 7 23:20:34.910062213 (13h40m56.654119064s) [upload=14088]: Got next segment. Inline: false Aug 7 23:20:34.910103244 (13h40m56.654160095s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:34.910108545 (13h40m56.654165405s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:04.641072069 (13h41m26.385128931s) [upload=14088]: Next segment err: context deadline exceeded Aug 7 23:21:04.641101566 (13h41m26.385158427s) [upload=14088]: Done uploading segments... Aug 7 23:21:04.641122347 (13h41m26.3851792s) [upload=14088]: Upload finished. err: context deadline exceeded Duration: 2.424017624s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [...................................XXX.XXXXXX..XX.X.........] duration: 896.642481ms FAIL (2025-08-07 23:27:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.340430421 (13h37m45.084487291s) [upload=14034]: Starting upload Aug 7 23:17:23.340585398 (13h37m45.084642265s) [upload=14034]: Done starting upload Aug 7 23:17:23.340743782 (13h37m45.084800644s) [upload=14034]: Uploading segments... Aug 7 23:17:23.340847668 (13h37m45.084904539s) [upload=14034]: Got next segment. Inline: true Aug 7 23:17:23.340857086 (13h37m45.084913958s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:17:23.340963706 (13h37m45.085020572s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.340973939 (13h37m45.085031006s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.340979371 (13h37m45.08503624s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.063547301 (13h38m14.807604164s) [upload=14034]: Done uploading segments... Aug 7 23:17:53.063575766 (13h38m14.807632627s) [upload=14034]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:01.235662578 (13h38m22.979719439s) [upload=14043]: Starting upload Aug 7 23:18:01.235769067 (13h38m22.97982594s) [upload=14043]: Done starting upload Aug 7 23:18:01.235873103 (13h38m22.979929971s) [upload=14043]: Uploading segments... Aug 7 23:18:01.235955349 (13h38m22.98001222s) [upload=14043]: Got next segment. Inline: true Aug 7 23:18:01.235976122 (13h38m22.980032984s) [upload=14043]: Waiting for error group managing segments... Aug 7 23:18:01.236032940 (13h38m22.980089801s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:01.236038848 (13h38m22.980095712s) [upload=14043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:01.236056240 (13h38m22.980113107s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:30.962606072 (13h38m52.706662932s) [upload=14043]: Done uploading segments... Aug 7 23:18:30.962671417 (13h38m52.706728279s) [upload=14043]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:52.160899256 (13h39m13.904956117s) [upload=14060]: Starting upload Aug 7 23:18:52.160970297 (13h39m13.905027157s) [upload=14060]: Done starting upload Aug 7 23:18:52.161065587 (13h39m13.905122439s) [upload=14060]: Uploading segments... Aug 7 23:18:52.161108065 (13h39m13.905164934s) [upload=14060]: Got next segment. Inline: true Aug 7 23:18:52.161113994 (13h39m13.905170844s) [upload=14060]: Waiting for error group managing segments... Aug 7 23:18:52.161171289 (13h39m13.90522814s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:52.161176753 (13h39m13.905233614s) [upload=14060]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:52.161180292 (13h39m13.905237155s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:52.299645097 (13h39m14.043701961s) [upload=14060]: Done uploading segments... Aug 7 23:18:52.299672156 (13h39m14.043729007s) [upload=14060]: Upload finished. err: <nil> Aug 7 23:19:29.899969190 (13h39m51.64402605s) [upload=14071]: Starting upload Aug 7 23:19:29.900023963 (13h39m51.644080812s) [upload=14071]: Done starting upload Aug 7 23:19:29.900097059 (13h39m51.644153908s) [upload=14071]: Uploading segments... Aug 7 23:19:29.900152738 (13h39m51.644209608s) [upload=14071]: Got next segment. Inline: true Aug 7 23:19:29.900161132 (13h39m51.644218002s) [upload=14071]: Waiting for error group managing segments... Aug 7 23:19:29.900212688 (13h39m51.64426955s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:29.900219311 (13h39m51.644276178s) [upload=14071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:29.900224731 (13h39m51.64428173s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:30.208671256 (13h39m51.952728106s) [upload=14071]: Done uploading segments... Aug 7 23:19:30.208715246 (13h39m51.952772096s) [upload=14071]: Upload finished. err: <nil> Aug 7 23:20:17.640185376 (13h40m39.384242236s) [upload=14084]: Starting upload Aug 7 23:20:17.640240796 (13h40m39.384297642s) [upload=14084]: Done starting upload Aug 7 23:20:17.640310076 (13h40m39.384366923s) [upload=14084]: Uploading segments... Aug 7 23:20:17.640362556 (13h40m39.384419425s) [upload=14084]: Got next segment. Inline: true Aug 7 23:20:17.640367233 (13h40m39.384424093s) [upload=14084]: Waiting for error group managing segments... Aug 7 23:20:17.640417842 (13h40m39.384474688s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:17.640422768 (13h40m39.384479628s) [upload=14084]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:20:17.640425589 (13h40m39.384482436s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:21.993094963 (13h40m43.737151826s) [upload=14084]: Done uploading segments... Aug 7 23:20:21.993114705 (13h40m43.737171555s) [upload=14084]: Upload finished. err: <nil> Aug 7 23:21:07.375939272 (13h41m29.119996122s) [upload=14096]: Starting upload Aug 7 23:21:07.376022371 (13h41m29.120079233s) [upload=14096]: Done starting upload Aug 7 23:21:07.376131644 (13h41m29.120188494s) [upload=14096]: Uploading segments... Aug 7 23:21:07.376228707 (13h41m29.120285578s) [upload=14096]: Got next segment. Inline: true Aug 7 23:21:07.376236415 (13h41m29.120293285s) [upload=14096]: Waiting for error group managing segments... Aug 7 23:21:07.376314696 (13h41m29.120371567s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:21:07.376322308 (13h41m29.120379199s) [upload=14096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:21:07.376327371 (13h41m29.120384242s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:37.104686329 (13h41m58.84874319s) [upload=14096]: Done uploading segments... Aug 7 23:21:37.104719076 (13h41m58.848775939s) [upload=14096]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:22:04.330992472 (13h42m26.075049325s) [upload=14111]: Starting upload Aug 7 23:22:04.331290477 (13h42m26.075347342s) [upload=14111]: Done starting upload Aug 7 23:22:04.331454134 (13h42m26.075510996s) [upload=14111]: Uploading segments... Aug 7 23:22:04.331559270 (13h42m26.075616144s) [upload=14111]: Got next segment. Inline: true Aug 7 23:22:04.331568404 (13h42m26.075625271s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:22:04.331647006 (13h42m26.075703873s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:22:04.331655756 (13h42m26.075712624s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:22:04.331661330 (13h42m26.075718197s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:22:04.632273425 (13h42m26.376330276s) [upload=14111]: Done uploading segments... Aug 7 23:22:04.632293274 (13h42m26.376350128s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:22:37.291949339 (13h42m59.03600619s) [upload=14122]: Starting upload Aug 7 23:22:37.292013250 (13h42m59.036070113s) [upload=14122]: Done starting upload Aug 7 23:22:37.292085244 (13h42m59.036142097s) [upload=14122]: Uploading segments... Aug 7 23:22:37.292207393 (13h42m59.036264263s) [upload=14122]: Got next segment. Inline: true Aug 7 23:22:37.292216268 (13h42m59.036273132s) [upload=14122]: Waiting for error group managing segments... Aug 7 23:22:37.292274925 (13h42m59.036331788s) [upload=14122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:22:37.292281983 (13h42m59.036338843s) [upload=14122]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:22:37.292288349 (13h42m59.036345217s) [upload=14122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:23:07.017248949 (13h43m28.761305819s) [upload=14122]: Done uploading segments... Aug 7 23:23:07.017274459 (13h43m28.761331331s) [upload=14122]: Upload finished. err: metaclient: context deadline exceeded Duration: 896.677704ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X.XX..X..............] duration: 2.755865836s FAIL (2025-08-07 23:27:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.755889576s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..........................................X.XXXXX.X.........] duration: 3.000300881s FAIL (2025-08-07 23:27:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.000328137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................................XXXXXX.XX..........] duration: 2.433467971s FAIL (2025-08-07 23:27:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.433505998s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [.....................................XXXXX.XX.XX.X..........] duration: 896.081162ms FAIL (2025-08-07 23:27:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 896.117021ms core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.........................................XXX...XX.X.........] duration: 1.135721521s FAIL (2025-08-07 23:27:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.135746337s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................................XXXXXXXXXX..........] duration: 2.920334666s FAIL (2025-08-07 23:27:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== upload Aug 7 23:18:25.127322319 (13h38m46.871379183s) [upload=14052]: Done starting upload Aug 7 23:18:25.127477364 (13h38m46.871534223s) [upload=14052]: Uploading segments... Aug 7 23:18:25.127943505 (13h38m46.872000369s) [upload=14052]: Got next segment. Inline: false Aug 7 23:18:25.128022972 (13h38m46.87207984s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:25.128032328 (13h38m46.872089195s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:44.923639390 (13h39m6.667696251s) [seg_pos={0 0}, upload=14052]: Begin upload segment... Aug 7 23:18:44.923694548 (13h39m6.667751411s) [seg_pos={0 0}, upload=14052]: Done begin upload segment. Aug 7 23:18:44.923719227 (13h39m6.667776093s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.923742940 (13h39m6.667799806s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924062820 (13h39m6.66811968s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924231371 (13h39m6.668288235s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924310412 (13h39m6.668367275s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924552072 (13h39m6.668608933s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924795545 (13h39m6.66885246s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925037101 (13h39m6.669093961s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925293050 (13h39m6.669349911s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925526381 (13h39m6.669583247s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925876037 (13h39m6.669932897s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926278285 (13h39m6.670335152s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926413382 (13h39m6.670470243s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926732421 (13h39m6.670789284s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927153074 (13h39m6.671209983s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927375228 (13h39m6.671432108s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:45.586845311 (13h39m7.330902181s) [upload=14052]: Next returned nil segment Aug 7 23:18:45.586873064 (13h39m7.330929934s) [upload=14052]: Waiting for error group managing segments... Aug 7 23:18:45.727354327 (13h39m7.471411192s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.767876801 (13h39m7.511933648s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.792222311 (13h39m7.536279182s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.836090391 (13h39m7.580147242s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.847910967 (13h39m7.591967821s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.874060717 (13h39m7.618117569s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.882796683 (13h39m7.626853529s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.883366192 (13h39m7.62742305s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.899253981 (13h39m7.643310835s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.905112812 (13h39m7.64916966s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.911636868 (13h39m7.655693715s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.934683670 (13h39m7.67874052s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959619489 (13h39m7.703676335s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959647576 (13h39m7.703704436s) [seg_pos={0 0}, upload=14052]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:45.959785097 (13h39m7.703841946s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959873513 (13h39m7.703930376s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959997706 (13h39m7.704054568s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection Aug 7 23:18:45.960040133 (13h39m7.70409698s) [seg_pos={0 0}, upload=14052]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:45.960143346 (13h39m7.704200223s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:45.960151195 (13h39m7.704208074s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:34.514397794 (13h40m56.258454651s) [upload=14052]: Done uploading segments... Aug 7 23:20:34.514434819 (13h40m56.258491687s) [upload=14052]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:20:34.909564598 (13h40m56.65362148s) [upload=14088]: Starting upload Aug 7 23:20:34.909657672 (13h40m56.653714552s) [upload=14088]: Done starting upload Aug 7 23:20:34.909836458 (13h40m56.653893325s) [upload=14088]: Uploading segments... Aug 7 23:20:34.910062213 (13h40m56.654119064s) [upload=14088]: Got next segment. Inline: false Aug 7 23:20:34.910103244 (13h40m56.654160095s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:34.910108545 (13h40m56.654165405s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:04.641072069 (13h41m26.385128931s) [upload=14088]: Next segment err: context deadline exceeded Aug 7 23:21:04.641101566 (13h41m26.385158427s) [upload=14088]: Done uploading segments... Aug 7 23:21:04.641122347 (13h41m26.3851792s) [upload=14088]: Upload finished. err: context deadline exceeded Duration: 2.920359895s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X...X.XX..........] duration: 3.271014701s FAIL (2025-08-07 23:27:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== inished. err: context deadline exceeded Aug 7 23:20:07.202369543 (13h40m28.946426394s) [upload=14082]: Starting upload Aug 7 23:20:07.202426657 (13h40m28.946483509s) [upload=14082]: Done starting upload Aug 7 23:20:07.202696950 (13h40m28.946753814s) [upload=14082]: Uploading segments... Aug 7 23:20:07.202761256 (13h40m28.94681811s) [upload=14082]: Got next segment. Inline: false Aug 7 23:20:07.202807046 (13h40m28.946863908s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:07.202814233 (13h40m28.946871102s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:20:45.722093118 (13h41m7.466149983s) [seg_pos={0 0}, upload=14082]: Begin upload segment... Aug 7 23:20:45.722177165 (13h41m7.466234018s) [seg_pos={0 0}, upload=14082]: Done begin upload segment. Aug 7 23:20:45.722218538 (13h41m7.4662754s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722246298 (13h41m7.46630316s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722330454 (13h41m7.466387316s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722658583 (13h41m7.466715506s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722717191 (13h41m7.46677405s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722989701 (13h41m7.467046568s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723061064 (13h41m7.467117927s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723458393 (13h41m7.467515259s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.729867582 (13h41m7.473924444s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.731063643 (13h41m7.475120512s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.732350031 (13h41m7.476406894s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.733989697 (13h41m7.478046556s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734727605 (13h41m7.478784474s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734786637 (13h41m7.478843504s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735460668 (13h41m7.479517532s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735486537 (13h41m7.479543401s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.736565943 (13h41m7.480622804s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.737386328 (13h41m7.48144319s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:46.299198843 (13h41m8.043255689s) [upload=14082]: Next returned nil segment Aug 7 23:20:46.299217870 (13h41m8.043274733s) [upload=14082]: Waiting for error group managing segments... Aug 7 23:20:46.464881860 (13h41m8.208938721s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.473775294 (13h41m8.217832164s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.474664669 (13h41m8.218721541s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.479688336 (13h41m8.223745197s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.485171804 (13h41m8.229228658s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.486836180 (13h41m8.230893032s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.496476947 (13h41m8.240533796s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.556494206 (13h41m8.300551081s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.628743021 (13h41m8.372799882s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.645987246 (13h41m8.390044106s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.666511132 (13h41m8.410567992s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.671756371 (13h41m8.41581322s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.934025878 (13h41m8.678082729s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970907150 (13h41m8.714964016s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970940415 (13h41m8.714997268s) [seg_pos={0 0}, upload=14082]: Segment reached optimal threshold of 14 pieces. Aug 7 23:20:46.971273207 (13h41m8.715330074s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MD2UDNXEmeGra9fvhEb8eGL2R6vGLHtSKLpSyX3H3tZCe1r5B): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971460895 (13h41m8.715517755s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971541847 (13h41m8.715598706s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJTNVtPFN48S9ZrayehLD2HcVckyJWLx2rtEvyH56W22R8Suq): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971599101 (13h41m8.715655962s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JeDjqgszkkq2asUBCL5oTvH6X21TnWkkmqGEEzBbz1YctdDXX): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971618718 (13h41m8.715675568s) [seg_pos={0 0}, upload=14082]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:20:46.971698846 (13h41m8.715755698s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:20:46.971704848 (13h41m8.71576171s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:07.219673652 (13h41m28.963730505s) [upload=14082]: Done uploading segments... Aug 7 23:21:07.219701493 (13h41m28.963758344s) [upload=14082]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.271059076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.412235286s FAIL (2025-08-07 23:27:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.412272845s core uplink eu1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 297.637257ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:27:26 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 297.663254ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X....X............X.............................] duration: 2.988732078s FAIL (2025-08-07 23:27:25 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 2.988764909s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X...X.XX.........] duration: 3.278809544s FAIL (2025-08-07 23:27:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== inished. err: context deadline exceeded Aug 7 23:20:07.202369543 (13h40m28.946426394s) [upload=14082]: Starting upload Aug 7 23:20:07.202426657 (13h40m28.946483509s) [upload=14082]: Done starting upload Aug 7 23:20:07.202696950 (13h40m28.946753814s) [upload=14082]: Uploading segments... Aug 7 23:20:07.202761256 (13h40m28.94681811s) [upload=14082]: Got next segment. Inline: false Aug 7 23:20:07.202807046 (13h40m28.946863908s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:07.202814233 (13h40m28.946871102s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:20:45.722093118 (13h41m7.466149983s) [seg_pos={0 0}, upload=14082]: Begin upload segment... Aug 7 23:20:45.722177165 (13h41m7.466234018s) [seg_pos={0 0}, upload=14082]: Done begin upload segment. Aug 7 23:20:45.722218538 (13h41m7.4662754s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722246298 (13h41m7.46630316s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722330454 (13h41m7.466387316s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722658583 (13h41m7.466715506s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722717191 (13h41m7.46677405s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722989701 (13h41m7.467046568s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723061064 (13h41m7.467117927s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723458393 (13h41m7.467515259s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.729867582 (13h41m7.473924444s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.731063643 (13h41m7.475120512s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.732350031 (13h41m7.476406894s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.733989697 (13h41m7.478046556s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734727605 (13h41m7.478784474s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734786637 (13h41m7.478843504s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735460668 (13h41m7.479517532s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735486537 (13h41m7.479543401s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.736565943 (13h41m7.480622804s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.737386328 (13h41m7.48144319s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:46.299198843 (13h41m8.043255689s) [upload=14082]: Next returned nil segment Aug 7 23:20:46.299217870 (13h41m8.043274733s) [upload=14082]: Waiting for error group managing segments... Aug 7 23:20:46.464881860 (13h41m8.208938721s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.473775294 (13h41m8.217832164s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.474664669 (13h41m8.218721541s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.479688336 (13h41m8.223745197s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.485171804 (13h41m8.229228658s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.486836180 (13h41m8.230893032s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.496476947 (13h41m8.240533796s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.556494206 (13h41m8.300551081s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.628743021 (13h41m8.372799882s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.645987246 (13h41m8.390044106s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.666511132 (13h41m8.410567992s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.671756371 (13h41m8.41581322s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.934025878 (13h41m8.678082729s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970907150 (13h41m8.714964016s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970940415 (13h41m8.714997268s) [seg_pos={0 0}, upload=14082]: Segment reached optimal threshold of 14 pieces. Aug 7 23:20:46.971273207 (13h41m8.715330074s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MD2UDNXEmeGra9fvhEb8eGL2R6vGLHtSKLpSyX3H3tZCe1r5B): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971460895 (13h41m8.715517755s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971541847 (13h41m8.715598706s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJTNVtPFN48S9ZrayehLD2HcVckyJWLx2rtEvyH56W22R8Suq): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971599101 (13h41m8.715655962s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JeDjqgszkkq2asUBCL5oTvH6X21TnWkkmqGEEzBbz1YctdDXX): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971618718 (13h41m8.715675568s) [seg_pos={0 0}, upload=14082]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:20:46.971698846 (13h41m8.715755698s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:20:46.971704848 (13h41m8.71576171s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:07.219673652 (13h41m28.963730505s) [upload=14082]: Done uploading segments... Aug 7 23:21:07.219701493 (13h41m28.963758344s) [upload=14082]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.278854796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXX.........] duration: 2.761707192s FAIL (2025-08-07 23:27:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== upload Aug 7 23:18:25.127322319 (13h38m46.871379183s) [upload=14052]: Done starting upload Aug 7 23:18:25.127477364 (13h38m46.871534223s) [upload=14052]: Uploading segments... Aug 7 23:18:25.127943505 (13h38m46.872000369s) [upload=14052]: Got next segment. Inline: false Aug 7 23:18:25.128022972 (13h38m46.87207984s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:25.128032328 (13h38m46.872089195s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:44.923639390 (13h39m6.667696251s) [seg_pos={0 0}, upload=14052]: Begin upload segment... Aug 7 23:18:44.923694548 (13h39m6.667751411s) [seg_pos={0 0}, upload=14052]: Done begin upload segment. Aug 7 23:18:44.923719227 (13h39m6.667776093s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.923742940 (13h39m6.667799806s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924062820 (13h39m6.66811968s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924231371 (13h39m6.668288235s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924310412 (13h39m6.668367275s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924552072 (13h39m6.668608933s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924795545 (13h39m6.66885246s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925037101 (13h39m6.669093961s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925293050 (13h39m6.669349911s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925526381 (13h39m6.669583247s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925876037 (13h39m6.669932897s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926278285 (13h39m6.670335152s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926413382 (13h39m6.670470243s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926732421 (13h39m6.670789284s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927153074 (13h39m6.671209983s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927375228 (13h39m6.671432108s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:45.586845311 (13h39m7.330902181s) [upload=14052]: Next returned nil segment Aug 7 23:18:45.586873064 (13h39m7.330929934s) [upload=14052]: Waiting for error group managing segments... Aug 7 23:18:45.727354327 (13h39m7.471411192s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.767876801 (13h39m7.511933648s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.792222311 (13h39m7.536279182s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.836090391 (13h39m7.580147242s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.847910967 (13h39m7.591967821s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.874060717 (13h39m7.618117569s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.882796683 (13h39m7.626853529s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.883366192 (13h39m7.62742305s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.899253981 (13h39m7.643310835s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.905112812 (13h39m7.64916966s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.911636868 (13h39m7.655693715s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.934683670 (13h39m7.67874052s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959619489 (13h39m7.703676335s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959647576 (13h39m7.703704436s) [seg_pos={0 0}, upload=14052]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:45.959785097 (13h39m7.703841946s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959873513 (13h39m7.703930376s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959997706 (13h39m7.704054568s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection Aug 7 23:18:45.960040133 (13h39m7.70409698s) [seg_pos={0 0}, upload=14052]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:45.960143346 (13h39m7.704200223s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:45.960151195 (13h39m7.704208074s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:34.514397794 (13h40m56.258454651s) [upload=14052]: Done uploading segments... Aug 7 23:20:34.514434819 (13h40m56.258491687s) [upload=14052]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:20:34.909564598 (13h40m56.65362148s) [upload=14088]: Starting upload Aug 7 23:20:34.909657672 (13h40m56.653714552s) [upload=14088]: Done starting upload Aug 7 23:20:34.909836458 (13h40m56.653893325s) [upload=14088]: Uploading segments... Aug 7 23:20:34.910062213 (13h40m56.654119064s) [upload=14088]: Got next segment. Inline: false Aug 7 23:20:34.910103244 (13h40m56.654160095s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:34.910108545 (13h40m56.654165405s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:04.641072069 (13h41m26.385128931s) [upload=14088]: Next segment err: context deadline exceeded Aug 7 23:21:04.641101566 (13h41m26.385158427s) [upload=14088]: Done uploading segments... Aug 7 23:21:04.641122347 (13h41m26.3851792s) [upload=14088]: Upload finished. err: context deadline exceeded Duration: 2.761741138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [....................................XXX.XXXXXX..XX.X........] duration: 844.9498ms FAIL (2025-08-07 23:27:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.340430421 (13h37m45.084487291s) [upload=14034]: Starting upload Aug 7 23:17:23.340585398 (13h37m45.084642265s) [upload=14034]: Done starting upload Aug 7 23:17:23.340743782 (13h37m45.084800644s) [upload=14034]: Uploading segments... Aug 7 23:17:23.340847668 (13h37m45.084904539s) [upload=14034]: Got next segment. Inline: true Aug 7 23:17:23.340857086 (13h37m45.084913958s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:17:23.340963706 (13h37m45.085020572s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.340973939 (13h37m45.085031006s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.340979371 (13h37m45.08503624s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.063547301 (13h38m14.807604164s) [upload=14034]: Done uploading segments... Aug 7 23:17:53.063575766 (13h38m14.807632627s) [upload=14034]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:01.235662578 (13h38m22.979719439s) [upload=14043]: Starting upload Aug 7 23:18:01.235769067 (13h38m22.97982594s) [upload=14043]: Done starting upload Aug 7 23:18:01.235873103 (13h38m22.979929971s) [upload=14043]: Uploading segments... Aug 7 23:18:01.235955349 (13h38m22.98001222s) [upload=14043]: Got next segment. Inline: true Aug 7 23:18:01.235976122 (13h38m22.980032984s) [upload=14043]: Waiting for error group managing segments... Aug 7 23:18:01.236032940 (13h38m22.980089801s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:01.236038848 (13h38m22.980095712s) [upload=14043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:01.236056240 (13h38m22.980113107s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:30.962606072 (13h38m52.706662932s) [upload=14043]: Done uploading segments... Aug 7 23:18:30.962671417 (13h38m52.706728279s) [upload=14043]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:52.160899256 (13h39m13.904956117s) [upload=14060]: Starting upload Aug 7 23:18:52.160970297 (13h39m13.905027157s) [upload=14060]: Done starting upload Aug 7 23:18:52.161065587 (13h39m13.905122439s) [upload=14060]: Uploading segments... Aug 7 23:18:52.161108065 (13h39m13.905164934s) [upload=14060]: Got next segment. Inline: true Aug 7 23:18:52.161113994 (13h39m13.905170844s) [upload=14060]: Waiting for error group managing segments... Aug 7 23:18:52.161171289 (13h39m13.90522814s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:52.161176753 (13h39m13.905233614s) [upload=14060]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:52.161180292 (13h39m13.905237155s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:52.299645097 (13h39m14.043701961s) [upload=14060]: Done uploading segments... Aug 7 23:18:52.299672156 (13h39m14.043729007s) [upload=14060]: Upload finished. err: <nil> Aug 7 23:19:29.899969190 (13h39m51.64402605s) [upload=14071]: Starting upload Aug 7 23:19:29.900023963 (13h39m51.644080812s) [upload=14071]: Done starting upload Aug 7 23:19:29.900097059 (13h39m51.644153908s) [upload=14071]: Uploading segments... Aug 7 23:19:29.900152738 (13h39m51.644209608s) [upload=14071]: Got next segment. Inline: true Aug 7 23:19:29.900161132 (13h39m51.644218002s) [upload=14071]: Waiting for error group managing segments... Aug 7 23:19:29.900212688 (13h39m51.64426955s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:29.900219311 (13h39m51.644276178s) [upload=14071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:29.900224731 (13h39m51.64428173s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:30.208671256 (13h39m51.952728106s) [upload=14071]: Done uploading segments... Aug 7 23:19:30.208715246 (13h39m51.952772096s) [upload=14071]: Upload finished. err: <nil> Aug 7 23:20:17.640185376 (13h40m39.384242236s) [upload=14084]: Starting upload Aug 7 23:20:17.640240796 (13h40m39.384297642s) [upload=14084]: Done starting upload Aug 7 23:20:17.640310076 (13h40m39.384366923s) [upload=14084]: Uploading segments... Aug 7 23:20:17.640362556 (13h40m39.384419425s) [upload=14084]: Got next segment. Inline: true Aug 7 23:20:17.640367233 (13h40m39.384424093s) [upload=14084]: Waiting for error group managing segments... Aug 7 23:20:17.640417842 (13h40m39.384474688s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:17.640422768 (13h40m39.384479628s) [upload=14084]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:20:17.640425589 (13h40m39.384482436s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:21.993094963 (13h40m43.737151826s) [upload=14084]: Done uploading segments... Aug 7 23:20:21.993114705 (13h40m43.737171555s) [upload=14084]: Upload finished. err: <nil> Aug 7 23:21:07.375939272 (13h41m29.119996122s) [upload=14096]: Starting upload Aug 7 23:21:07.376022371 (13h41m29.120079233s) [upload=14096]: Done starting upload Aug 7 23:21:07.376131644 (13h41m29.120188494s) [upload=14096]: Uploading segments... Aug 7 23:21:07.376228707 (13h41m29.120285578s) [upload=14096]: Got next segment. Inline: true Aug 7 23:21:07.376236415 (13h41m29.120293285s) [upload=14096]: Waiting for error group managing segments... Aug 7 23:21:07.376314696 (13h41m29.120371567s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:21:07.376322308 (13h41m29.120379199s) [upload=14096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:21:07.376327371 (13h41m29.120384242s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:37.104686329 (13h41m58.84874319s) [upload=14096]: Done uploading segments... Aug 7 23:21:37.104719076 (13h41m58.848775939s) [upload=14096]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:22:04.330992472 (13h42m26.075049325s) [upload=14111]: Starting upload Aug 7 23:22:04.331290477 (13h42m26.075347342s) [upload=14111]: Done starting upload Aug 7 23:22:04.331454134 (13h42m26.075510996s) [upload=14111]: Uploading segments... Aug 7 23:22:04.331559270 (13h42m26.075616144s) [upload=14111]: Got next segment. Inline: true Aug 7 23:22:04.331568404 (13h42m26.075625271s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:22:04.331647006 (13h42m26.075703873s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:22:04.331655756 (13h42m26.075712624s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:22:04.331661330 (13h42m26.075718197s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:22:04.632273425 (13h42m26.376330276s) [upload=14111]: Done uploading segments... Aug 7 23:22:04.632293274 (13h42m26.376350128s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:22:37.291949339 (13h42m59.03600619s) [upload=14122]: Starting upload Aug 7 23:22:37.292013250 (13h42m59.036070113s) [upload=14122]: Done starting upload Aug 7 23:22:37.292085244 (13h42m59.036142097s) [upload=14122]: Uploading segments... Aug 7 23:22:37.292207393 (13h42m59.036264263s) [upload=14122]: Got next segment. Inline: true Aug 7 23:22:37.292216268 (13h42m59.036273132s) [upload=14122]: Waiting for error group managing segments... Aug 7 23:22:37.292274925 (13h42m59.036331788s) [upload=14122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:22:37.292281983 (13h42m59.036338843s) [upload=14122]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:22:37.292288349 (13h42m59.036345217s) [upload=14122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:23:07.017248949 (13h43m28.761305819s) [upload=14122]: Done uploading segments... Aug 7 23:23:07.017274459 (13h43m28.761331331s) [upload=14122]: Upload finished. err: metaclient: context deadline exceeded Duration: 844.984056ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [.......................................XX..XXXXX............] duration: 838.678237ms FAIL (2025-08-07 23:27:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:11:40.667902567 (13h32m2.411959413s) [upload=13933]: Starting upload Aug 7 23:11:40.667961940 (13h32m2.412018788s) [upload=13933]: Done starting upload Aug 7 23:11:40.668020524 (13h32m2.41207737s) [upload=13933]: Uploading segments... Aug 7 23:11:40.668075247 (13h32m2.412132113s) [upload=13933]: Got next segment. Inline: true Aug 7 23:11:40.668080664 (13h32m2.412137525s) [upload=13933]: Waiting for error group managing segments... Aug 7 23:11:40.668121190 (13h32m2.412178036s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:40.668125756 (13h32m2.412182616s) [upload=13933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:40.668128431 (13h32m2.412185291s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.394243254 (13h33m2.138300118s) [upload=13933]: Done uploading segments... Aug 7 23:12:40.394270270 (13h33m2.138327132s) [upload=13933]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:40.798630406 (13h33m2.542687267s) [upload=13953]: Starting upload Aug 7 23:12:40.798725809 (13h33m2.54278266s) [upload=13953]: Done starting upload Aug 7 23:12:40.798811946 (13h33m2.542868796s) [upload=13953]: Uploading segments... Aug 7 23:12:40.798867069 (13h33m2.542923932s) [upload=13953]: Got next segment. Inline: true Aug 7 23:12:40.798873108 (13h33m2.542929959s) [upload=13953]: Waiting for error group managing segments... Aug 7 23:12:40.798918573 (13h33m2.542975422s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:40.798924603 (13h33m2.542981463s) [upload=13953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:40.798939855 (13h33m2.542996716s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.942390421 (13h33m2.686447272s) [upload=13953]: Done uploading segments... Aug 7 23:12:40.942410212 (13h33m2.686467062s) [upload=13953]: Upload finished. err: <nil> Aug 7 23:13:18.137108974 (13h33m39.881165825s) [upload=13967]: Starting upload Aug 7 23:13:18.137180408 (13h33m39.881237292s) [upload=13967]: Done starting upload Aug 7 23:13:18.137258925 (13h33m39.881315786s) [upload=13967]: Uploading segments... Aug 7 23:13:18.137309635 (13h33m39.881366505s) [upload=13967]: Got next segment. Inline: true Aug 7 23:13:18.137314511 (13h33m39.881371371s) [upload=13967]: Waiting for error group managing segments... Aug 7 23:13:18.137374772 (13h33m39.881431633s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.137380412 (13h33m39.881437273s) [upload=13967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.137383733 (13h33m39.881440593s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.301480519 (13h33m40.045537379s) [upload=13967]: Done uploading segments... Aug 7 23:13:18.301498918 (13h33m40.045555767s) [upload=13967]: Upload finished. err: <nil> Aug 7 23:14:13.169029066 (13h34m34.913085913s) [upload=13980]: Starting upload Aug 7 23:14:13.169095254 (13h34m34.913152101s) [upload=13980]: Done starting upload Aug 7 23:14:13.169153338 (13h34m34.913210184s) [upload=13980]: Uploading segments... Aug 7 23:14:13.169225662 (13h34m34.913282508s) [upload=13980]: Got next segment. Inline: true Aug 7 23:14:13.169230885 (13h34m34.913287744s) [upload=13980]: Waiting for error group managing segments... Aug 7 23:14:13.169272521 (13h34m34.913329381s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:13.169288952 (13h34m34.913345813s) [upload=13980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:13.169291623 (13h34m34.913348483s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:12.895960606 (13h35m34.640017467s) [upload=13980]: Done uploading segments... Aug 7 23:15:12.895983608 (13h35m34.640040456s) [upload=13980]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:13.308146067 (13h35m35.052202918s) [upload=13997]: Starting upload Aug 7 23:15:13.308219271 (13h35m35.052276134s) [upload=13997]: Done starting upload Aug 7 23:15:13.308297319 (13h35m35.052354171s) [upload=13997]: Uploading segments... Aug 7 23:15:13.308360203 (13h35m35.052417072s) [upload=13997]: Got next segment. Inline: true Aug 7 23:15:13.308365548 (13h35m35.052422408s) [upload=13997]: Waiting for error group managing segments... Aug 7 23:15:13.308409794 (13h35m35.052466657s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:13.308426102 (13h35m35.052482963s) [upload=13997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:13.308429725 (13h35m35.052486585s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:13.040589970 (13h36m34.784646843s) [upload=13997]: Done uploading segments... Aug 7 23:16:13.040614428 (13h36m34.784671274s) [upload=13997]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:20.767866726 (13h36m42.511923577s) [upload=14018]: Starting upload Aug 7 23:16:20.767927544 (13h36m42.511984398s) [upload=14018]: Done starting upload Aug 7 23:16:20.767993090 (13h36m42.512049942s) [upload=14018]: Uploading segments... Aug 7 23:16:20.768055054 (13h36m42.512111926s) [upload=14018]: Got next segment. Inline: true Aug 7 23:16:20.768061500 (13h36m42.512118351s) [upload=14018]: Waiting for error group managing segments... Aug 7 23:16:20.768130604 (13h36m42.512187577s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:20.768139327 (13h36m42.512196192s) [upload=14018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:20.768144987 (13h36m42.512201849s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:20.496845379 (13h37m42.240902231s) [upload=14018]: Done uploading segments... Aug 7 23:17:20.496873778 (13h37m42.240930632s) [upload=14018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:20.916777280 (13h37m42.660834146s) [upload=14032]: Starting upload Aug 7 23:17:20.916877286 (13h37m42.660934149s) [upload=14032]: Done starting upload Aug 7 23:17:20.917002557 (13h37m42.661059408s) [upload=14032]: Uploading segments... Aug 7 23:17:20.917058206 (13h37m42.661115077s) [upload=14032]: Got next segment. Inline: true Aug 7 23:17:20.917064317 (13h37m42.661121178s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:17:20.917116347 (13h37m42.661173214s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:20.917123150 (13h37m42.661180014s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:20.917127779 (13h37m42.661184648s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:20.640884349 (13h38m42.384941214s) [upload=14032]: Done uploading segments... Aug 7 23:18:20.640914749 (13h38m42.384971617s) [upload=14032]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:21.050365938 (13h38m42.79442281s) [upload=14051]: Starting upload Aug 7 23:18:21.050484720 (13h38m42.794541587s) [upload=14051]: Done starting upload Aug 7 23:18:21.050589902 (13h38m42.794646752s) [upload=14051]: Uploading segments... Aug 7 23:18:21.050663797 (13h38m42.794720667s) [upload=14051]: Got next segment. Inline: true Aug 7 23:18:21.050670451 (13h38m42.794727311s) [upload=14051]: Waiting for error group managing segments... Aug 7 23:18:21.050719413 (13h38m42.794776275s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:21.050725163 (13h38m42.794782023s) [upload=14051]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:21.050728952 (13h38m42.794785811s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:20.780076504 (13h39m42.524133366s) [upload=14051]: Done uploading segments... Aug 7 23:19:20.780099469 (13h39m42.52415633s) [upload=14051]: Upload finished. err: metaclient: context deadline exceeded Duration: 838.708421ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [..........................................XXX...XX.X........] duration: 1.084488121s FAIL (2025-08-07 23:27:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.084516803s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.750783351s FAIL (2025-08-07 23:27:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.750836684s core uplink eu1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 319.11587ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:27:00 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 319.141125ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................................XXXXXX.XX.........] duration: 2.220712732s FAIL (2025-08-07 23:26:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.220736459s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X....X............X............................] duration: 3.442787554s FAIL (2025-08-07 23:26:54 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 3.442835556s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................X.XX..X.............] duration: 2.741984425s FAIL (2025-08-07 23:26:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.742017232s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXX........] duration: 4.085336201s FAIL (2025-08-07 23:26:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== upload Aug 7 23:18:25.127322319 (13h38m46.871379183s) [upload=14052]: Done starting upload Aug 7 23:18:25.127477364 (13h38m46.871534223s) [upload=14052]: Uploading segments... Aug 7 23:18:25.127943505 (13h38m46.872000369s) [upload=14052]: Got next segment. Inline: false Aug 7 23:18:25.128022972 (13h38m46.87207984s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:25.128032328 (13h38m46.872089195s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:44.923639390 (13h39m6.667696251s) [seg_pos={0 0}, upload=14052]: Begin upload segment... Aug 7 23:18:44.923694548 (13h39m6.667751411s) [seg_pos={0 0}, upload=14052]: Done begin upload segment. Aug 7 23:18:44.923719227 (13h39m6.667776093s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.923742940 (13h39m6.667799806s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924062820 (13h39m6.66811968s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924231371 (13h39m6.668288235s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924310412 (13h39m6.668367275s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924552072 (13h39m6.668608933s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924795545 (13h39m6.66885246s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925037101 (13h39m6.669093961s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925293050 (13h39m6.669349911s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925526381 (13h39m6.669583247s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925876037 (13h39m6.669932897s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926278285 (13h39m6.670335152s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926413382 (13h39m6.670470243s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926732421 (13h39m6.670789284s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927153074 (13h39m6.671209983s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927375228 (13h39m6.671432108s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:45.586845311 (13h39m7.330902181s) [upload=14052]: Next returned nil segment Aug 7 23:18:45.586873064 (13h39m7.330929934s) [upload=14052]: Waiting for error group managing segments... Aug 7 23:18:45.727354327 (13h39m7.471411192s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.767876801 (13h39m7.511933648s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.792222311 (13h39m7.536279182s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.836090391 (13h39m7.580147242s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.847910967 (13h39m7.591967821s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.874060717 (13h39m7.618117569s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.882796683 (13h39m7.626853529s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.883366192 (13h39m7.62742305s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.899253981 (13h39m7.643310835s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.905112812 (13h39m7.64916966s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.911636868 (13h39m7.655693715s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.934683670 (13h39m7.67874052s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959619489 (13h39m7.703676335s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959647576 (13h39m7.703704436s) [seg_pos={0 0}, upload=14052]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:45.959785097 (13h39m7.703841946s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959873513 (13h39m7.703930376s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959997706 (13h39m7.704054568s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection Aug 7 23:18:45.960040133 (13h39m7.70409698s) [seg_pos={0 0}, upload=14052]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:45.960143346 (13h39m7.704200223s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:45.960151195 (13h39m7.704208074s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:34.514397794 (13h40m56.258454651s) [upload=14052]: Done uploading segments... Aug 7 23:20:34.514434819 (13h40m56.258491687s) [upload=14052]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:20:34.909564598 (13h40m56.65362148s) [upload=14088]: Starting upload Aug 7 23:20:34.909657672 (13h40m56.653714552s) [upload=14088]: Done starting upload Aug 7 23:20:34.909836458 (13h40m56.653893325s) [upload=14088]: Uploading segments... Aug 7 23:20:34.910062213 (13h40m56.654119064s) [upload=14088]: Got next segment. Inline: false Aug 7 23:20:34.910103244 (13h40m56.654160095s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:34.910108545 (13h40m56.654165405s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:04.641072069 (13h41m26.385128931s) [upload=14088]: Next segment err: context deadline exceeded Aug 7 23:21:04.641101566 (13h41m26.385158427s) [upload=14088]: Done uploading segments... Aug 7 23:21:04.641122347 (13h41m26.3851792s) [upload=14088]: Upload finished. err: context deadline exceeded Duration: 4.085369747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................X...X.XX........] duration: 3.045693756s FAIL (2025-08-07 23:26:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== inished. err: context deadline exceeded Aug 7 23:20:07.202369543 (13h40m28.946426394s) [upload=14082]: Starting upload Aug 7 23:20:07.202426657 (13h40m28.946483509s) [upload=14082]: Done starting upload Aug 7 23:20:07.202696950 (13h40m28.946753814s) [upload=14082]: Uploading segments... Aug 7 23:20:07.202761256 (13h40m28.94681811s) [upload=14082]: Got next segment. Inline: false Aug 7 23:20:07.202807046 (13h40m28.946863908s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:07.202814233 (13h40m28.946871102s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:20:45.722093118 (13h41m7.466149983s) [seg_pos={0 0}, upload=14082]: Begin upload segment... Aug 7 23:20:45.722177165 (13h41m7.466234018s) [seg_pos={0 0}, upload=14082]: Done begin upload segment. Aug 7 23:20:45.722218538 (13h41m7.4662754s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722246298 (13h41m7.46630316s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722330454 (13h41m7.466387316s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722658583 (13h41m7.466715506s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722717191 (13h41m7.46677405s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722989701 (13h41m7.467046568s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723061064 (13h41m7.467117927s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723458393 (13h41m7.467515259s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.729867582 (13h41m7.473924444s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.731063643 (13h41m7.475120512s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.732350031 (13h41m7.476406894s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.733989697 (13h41m7.478046556s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734727605 (13h41m7.478784474s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734786637 (13h41m7.478843504s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735460668 (13h41m7.479517532s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735486537 (13h41m7.479543401s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.736565943 (13h41m7.480622804s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.737386328 (13h41m7.48144319s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:46.299198843 (13h41m8.043255689s) [upload=14082]: Next returned nil segment Aug 7 23:20:46.299217870 (13h41m8.043274733s) [upload=14082]: Waiting for error group managing segments... Aug 7 23:20:46.464881860 (13h41m8.208938721s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.473775294 (13h41m8.217832164s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.474664669 (13h41m8.218721541s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.479688336 (13h41m8.223745197s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.485171804 (13h41m8.229228658s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.486836180 (13h41m8.230893032s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.496476947 (13h41m8.240533796s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.556494206 (13h41m8.300551081s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.628743021 (13h41m8.372799882s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.645987246 (13h41m8.390044106s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.666511132 (13h41m8.410567992s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.671756371 (13h41m8.41581322s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.934025878 (13h41m8.678082729s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970907150 (13h41m8.714964016s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970940415 (13h41m8.714997268s) [seg_pos={0 0}, upload=14082]: Segment reached optimal threshold of 14 pieces. Aug 7 23:20:46.971273207 (13h41m8.715330074s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MD2UDNXEmeGra9fvhEb8eGL2R6vGLHtSKLpSyX3H3tZCe1r5B): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971460895 (13h41m8.715517755s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971541847 (13h41m8.715598706s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJTNVtPFN48S9ZrayehLD2HcVckyJWLx2rtEvyH56W22R8Suq): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971599101 (13h41m8.715655962s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JeDjqgszkkq2asUBCL5oTvH6X21TnWkkmqGEEzBbz1YctdDXX): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971618718 (13h41m8.715675568s) [seg_pos={0 0}, upload=14082]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:20:46.971698846 (13h41m8.715755698s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:20:46.971704848 (13h41m8.71576171s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:07.219673652 (13h41m28.963730505s) [upload=14082]: Done uploading segments... Aug 7 23:21:07.219701493 (13h41m28.963758344s) [upload=14082]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.045736818s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...........................................X.XXXXX.X........] duration: 3.108078843s FAIL (2025-08-07 23:26:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.108146174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [......................................XXXXX.XX.XX.X.........] duration: 931.669636ms FAIL (2025-08-07 23:26:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 931.706305ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.....................................XXX.XXXXXX..XX.X.......] duration: 888.791686ms FAIL (2025-08-07 23:26:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.340430421 (13h37m45.084487291s) [upload=14034]: Starting upload Aug 7 23:17:23.340585398 (13h37m45.084642265s) [upload=14034]: Done starting upload Aug 7 23:17:23.340743782 (13h37m45.084800644s) [upload=14034]: Uploading segments... Aug 7 23:17:23.340847668 (13h37m45.084904539s) [upload=14034]: Got next segment. Inline: true Aug 7 23:17:23.340857086 (13h37m45.084913958s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:17:23.340963706 (13h37m45.085020572s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.340973939 (13h37m45.085031006s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.340979371 (13h37m45.08503624s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.063547301 (13h38m14.807604164s) [upload=14034]: Done uploading segments... Aug 7 23:17:53.063575766 (13h38m14.807632627s) [upload=14034]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:01.235662578 (13h38m22.979719439s) [upload=14043]: Starting upload Aug 7 23:18:01.235769067 (13h38m22.97982594s) [upload=14043]: Done starting upload Aug 7 23:18:01.235873103 (13h38m22.979929971s) [upload=14043]: Uploading segments... Aug 7 23:18:01.235955349 (13h38m22.98001222s) [upload=14043]: Got next segment. Inline: true Aug 7 23:18:01.235976122 (13h38m22.980032984s) [upload=14043]: Waiting for error group managing segments... Aug 7 23:18:01.236032940 (13h38m22.980089801s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:01.236038848 (13h38m22.980095712s) [upload=14043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:01.236056240 (13h38m22.980113107s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:30.962606072 (13h38m52.706662932s) [upload=14043]: Done uploading segments... Aug 7 23:18:30.962671417 (13h38m52.706728279s) [upload=14043]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:52.160899256 (13h39m13.904956117s) [upload=14060]: Starting upload Aug 7 23:18:52.160970297 (13h39m13.905027157s) [upload=14060]: Done starting upload Aug 7 23:18:52.161065587 (13h39m13.905122439s) [upload=14060]: Uploading segments... Aug 7 23:18:52.161108065 (13h39m13.905164934s) [upload=14060]: Got next segment. Inline: true Aug 7 23:18:52.161113994 (13h39m13.905170844s) [upload=14060]: Waiting for error group managing segments... Aug 7 23:18:52.161171289 (13h39m13.90522814s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:52.161176753 (13h39m13.905233614s) [upload=14060]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:52.161180292 (13h39m13.905237155s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:52.299645097 (13h39m14.043701961s) [upload=14060]: Done uploading segments... Aug 7 23:18:52.299672156 (13h39m14.043729007s) [upload=14060]: Upload finished. err: <nil> Aug 7 23:19:29.899969190 (13h39m51.64402605s) [upload=14071]: Starting upload Aug 7 23:19:29.900023963 (13h39m51.644080812s) [upload=14071]: Done starting upload Aug 7 23:19:29.900097059 (13h39m51.644153908s) [upload=14071]: Uploading segments... Aug 7 23:19:29.900152738 (13h39m51.644209608s) [upload=14071]: Got next segment. Inline: true Aug 7 23:19:29.900161132 (13h39m51.644218002s) [upload=14071]: Waiting for error group managing segments... Aug 7 23:19:29.900212688 (13h39m51.64426955s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:29.900219311 (13h39m51.644276178s) [upload=14071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:29.900224731 (13h39m51.64428173s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:30.208671256 (13h39m51.952728106s) [upload=14071]: Done uploading segments... Aug 7 23:19:30.208715246 (13h39m51.952772096s) [upload=14071]: Upload finished. err: <nil> Aug 7 23:20:17.640185376 (13h40m39.384242236s) [upload=14084]: Starting upload Aug 7 23:20:17.640240796 (13h40m39.384297642s) [upload=14084]: Done starting upload Aug 7 23:20:17.640310076 (13h40m39.384366923s) [upload=14084]: Uploading segments... Aug 7 23:20:17.640362556 (13h40m39.384419425s) [upload=14084]: Got next segment. Inline: true Aug 7 23:20:17.640367233 (13h40m39.384424093s) [upload=14084]: Waiting for error group managing segments... Aug 7 23:20:17.640417842 (13h40m39.384474688s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:17.640422768 (13h40m39.384479628s) [upload=14084]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:20:17.640425589 (13h40m39.384482436s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:21.993094963 (13h40m43.737151826s) [upload=14084]: Done uploading segments... Aug 7 23:20:21.993114705 (13h40m43.737171555s) [upload=14084]: Upload finished. err: <nil> Aug 7 23:21:07.375939272 (13h41m29.119996122s) [upload=14096]: Starting upload Aug 7 23:21:07.376022371 (13h41m29.120079233s) [upload=14096]: Done starting upload Aug 7 23:21:07.376131644 (13h41m29.120188494s) [upload=14096]: Uploading segments... Aug 7 23:21:07.376228707 (13h41m29.120285578s) [upload=14096]: Got next segment. Inline: true Aug 7 23:21:07.376236415 (13h41m29.120293285s) [upload=14096]: Waiting for error group managing segments... Aug 7 23:21:07.376314696 (13h41m29.120371567s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:21:07.376322308 (13h41m29.120379199s) [upload=14096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:21:07.376327371 (13h41m29.120384242s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:37.104686329 (13h41m58.84874319s) [upload=14096]: Done uploading segments... Aug 7 23:21:37.104719076 (13h41m58.848775939s) [upload=14096]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:22:04.330992472 (13h42m26.075049325s) [upload=14111]: Starting upload Aug 7 23:22:04.331290477 (13h42m26.075347342s) [upload=14111]: Done starting upload Aug 7 23:22:04.331454134 (13h42m26.075510996s) [upload=14111]: Uploading segments... Aug 7 23:22:04.331559270 (13h42m26.075616144s) [upload=14111]: Got next segment. Inline: true Aug 7 23:22:04.331568404 (13h42m26.075625271s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:22:04.331647006 (13h42m26.075703873s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:22:04.331655756 (13h42m26.075712624s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:22:04.331661330 (13h42m26.075718197s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:22:04.632273425 (13h42m26.376330276s) [upload=14111]: Done uploading segments... Aug 7 23:22:04.632293274 (13h42m26.376350128s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:22:37.291949339 (13h42m59.03600619s) [upload=14122]: Starting upload Aug 7 23:22:37.292013250 (13h42m59.036070113s) [upload=14122]: Done starting upload Aug 7 23:22:37.292085244 (13h42m59.036142097s) [upload=14122]: Uploading segments... Aug 7 23:22:37.292207393 (13h42m59.036264263s) [upload=14122]: Got next segment. Inline: true Aug 7 23:22:37.292216268 (13h42m59.036273132s) [upload=14122]: Waiting for error group managing segments... Aug 7 23:22:37.292274925 (13h42m59.036331788s) [upload=14122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:22:37.292281983 (13h42m59.036338843s) [upload=14122]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:22:37.292288349 (13h42m59.036345217s) [upload=14122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:23:07.017248949 (13h43m28.761305819s) [upload=14122]: Done uploading segments... Aug 7 23:23:07.017274459 (13h43m28.761331331s) [upload=14122]: Upload finished. err: metaclient: context deadline exceeded Duration: 888.819872ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X.XX..X............] duration: 2.583773614s FAIL (2025-08-07 23:26:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.583797033s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [...........................................XXX...XX.X.......] duration: 1.139476419s FAIL (2025-08-07 23:26:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.139500821s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [........................................XX..XXXXX...........] duration: 860.703493ms FAIL (2025-08-07 23:26:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:11:40.667902567 (13h32m2.411959413s) [upload=13933]: Starting upload Aug 7 23:11:40.667961940 (13h32m2.412018788s) [upload=13933]: Done starting upload Aug 7 23:11:40.668020524 (13h32m2.41207737s) [upload=13933]: Uploading segments... Aug 7 23:11:40.668075247 (13h32m2.412132113s) [upload=13933]: Got next segment. Inline: true Aug 7 23:11:40.668080664 (13h32m2.412137525s) [upload=13933]: Waiting for error group managing segments... Aug 7 23:11:40.668121190 (13h32m2.412178036s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:40.668125756 (13h32m2.412182616s) [upload=13933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:40.668128431 (13h32m2.412185291s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.394243254 (13h33m2.138300118s) [upload=13933]: Done uploading segments... Aug 7 23:12:40.394270270 (13h33m2.138327132s) [upload=13933]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:40.798630406 (13h33m2.542687267s) [upload=13953]: Starting upload Aug 7 23:12:40.798725809 (13h33m2.54278266s) [upload=13953]: Done starting upload Aug 7 23:12:40.798811946 (13h33m2.542868796s) [upload=13953]: Uploading segments... Aug 7 23:12:40.798867069 (13h33m2.542923932s) [upload=13953]: Got next segment. Inline: true Aug 7 23:12:40.798873108 (13h33m2.542929959s) [upload=13953]: Waiting for error group managing segments... Aug 7 23:12:40.798918573 (13h33m2.542975422s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:40.798924603 (13h33m2.542981463s) [upload=13953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:40.798939855 (13h33m2.542996716s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.942390421 (13h33m2.686447272s) [upload=13953]: Done uploading segments... Aug 7 23:12:40.942410212 (13h33m2.686467062s) [upload=13953]: Upload finished. err: <nil> Aug 7 23:13:18.137108974 (13h33m39.881165825s) [upload=13967]: Starting upload Aug 7 23:13:18.137180408 (13h33m39.881237292s) [upload=13967]: Done starting upload Aug 7 23:13:18.137258925 (13h33m39.881315786s) [upload=13967]: Uploading segments... Aug 7 23:13:18.137309635 (13h33m39.881366505s) [upload=13967]: Got next segment. Inline: true Aug 7 23:13:18.137314511 (13h33m39.881371371s) [upload=13967]: Waiting for error group managing segments... Aug 7 23:13:18.137374772 (13h33m39.881431633s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.137380412 (13h33m39.881437273s) [upload=13967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.137383733 (13h33m39.881440593s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.301480519 (13h33m40.045537379s) [upload=13967]: Done uploading segments... Aug 7 23:13:18.301498918 (13h33m40.045555767s) [upload=13967]: Upload finished. err: <nil> Aug 7 23:14:13.169029066 (13h34m34.913085913s) [upload=13980]: Starting upload Aug 7 23:14:13.169095254 (13h34m34.913152101s) [upload=13980]: Done starting upload Aug 7 23:14:13.169153338 (13h34m34.913210184s) [upload=13980]: Uploading segments... Aug 7 23:14:13.169225662 (13h34m34.913282508s) [upload=13980]: Got next segment. Inline: true Aug 7 23:14:13.169230885 (13h34m34.913287744s) [upload=13980]: Waiting for error group managing segments... Aug 7 23:14:13.169272521 (13h34m34.913329381s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:13.169288952 (13h34m34.913345813s) [upload=13980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:13.169291623 (13h34m34.913348483s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:12.895960606 (13h35m34.640017467s) [upload=13980]: Done uploading segments... Aug 7 23:15:12.895983608 (13h35m34.640040456s) [upload=13980]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:13.308146067 (13h35m35.052202918s) [upload=13997]: Starting upload Aug 7 23:15:13.308219271 (13h35m35.052276134s) [upload=13997]: Done starting upload Aug 7 23:15:13.308297319 (13h35m35.052354171s) [upload=13997]: Uploading segments... Aug 7 23:15:13.308360203 (13h35m35.052417072s) [upload=13997]: Got next segment. Inline: true Aug 7 23:15:13.308365548 (13h35m35.052422408s) [upload=13997]: Waiting for error group managing segments... Aug 7 23:15:13.308409794 (13h35m35.052466657s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:13.308426102 (13h35m35.052482963s) [upload=13997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:13.308429725 (13h35m35.052486585s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:13.040589970 (13h36m34.784646843s) [upload=13997]: Done uploading segments... Aug 7 23:16:13.040614428 (13h36m34.784671274s) [upload=13997]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:20.767866726 (13h36m42.511923577s) [upload=14018]: Starting upload Aug 7 23:16:20.767927544 (13h36m42.511984398s) [upload=14018]: Done starting upload Aug 7 23:16:20.767993090 (13h36m42.512049942s) [upload=14018]: Uploading segments... Aug 7 23:16:20.768055054 (13h36m42.512111926s) [upload=14018]: Got next segment. Inline: true Aug 7 23:16:20.768061500 (13h36m42.512118351s) [upload=14018]: Waiting for error group managing segments... Aug 7 23:16:20.768130604 (13h36m42.512187577s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:20.768139327 (13h36m42.512196192s) [upload=14018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:20.768144987 (13h36m42.512201849s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:20.496845379 (13h37m42.240902231s) [upload=14018]: Done uploading segments... Aug 7 23:17:20.496873778 (13h37m42.240930632s) [upload=14018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:20.916777280 (13h37m42.660834146s) [upload=14032]: Starting upload Aug 7 23:17:20.916877286 (13h37m42.660934149s) [upload=14032]: Done starting upload Aug 7 23:17:20.917002557 (13h37m42.661059408s) [upload=14032]: Uploading segments... Aug 7 23:17:20.917058206 (13h37m42.661115077s) [upload=14032]: Got next segment. Inline: true Aug 7 23:17:20.917064317 (13h37m42.661121178s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:17:20.917116347 (13h37m42.661173214s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:20.917123150 (13h37m42.661180014s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:20.917127779 (13h37m42.661184648s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:20.640884349 (13h38m42.384941214s) [upload=14032]: Done uploading segments... Aug 7 23:18:20.640914749 (13h38m42.384971617s) [upload=14032]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:21.050365938 (13h38m42.79442281s) [upload=14051]: Starting upload Aug 7 23:18:21.050484720 (13h38m42.794541587s) [upload=14051]: Done starting upload Aug 7 23:18:21.050589902 (13h38m42.794646752s) [upload=14051]: Uploading segments... Aug 7 23:18:21.050663797 (13h38m42.794720667s) [upload=14051]: Got next segment. Inline: true Aug 7 23:18:21.050670451 (13h38m42.794727311s) [upload=14051]: Waiting for error group managing segments... Aug 7 23:18:21.050719413 (13h38m42.794776275s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:21.050725163 (13h38m42.794782023s) [upload=14051]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:21.050728952 (13h38m42.794785811s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:20.780076504 (13h39m42.524133366s) [upload=14051]: Done uploading segments... Aug 7 23:19:20.780099469 (13h39m42.52415633s) [upload=14051]: Upload finished. err: metaclient: context deadline exceeded Duration: 860.742788ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [......................................XXX.XXXXXX..XX.X......] duration: 985.082817ms FAIL (2025-08-07 23:26:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.340430421 (13h37m45.084487291s) [upload=14034]: Starting upload Aug 7 23:17:23.340585398 (13h37m45.084642265s) [upload=14034]: Done starting upload Aug 7 23:17:23.340743782 (13h37m45.084800644s) [upload=14034]: Uploading segments... Aug 7 23:17:23.340847668 (13h37m45.084904539s) [upload=14034]: Got next segment. Inline: true Aug 7 23:17:23.340857086 (13h37m45.084913958s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:17:23.340963706 (13h37m45.085020572s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.340973939 (13h37m45.085031006s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.340979371 (13h37m45.08503624s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.063547301 (13h38m14.807604164s) [upload=14034]: Done uploading segments... Aug 7 23:17:53.063575766 (13h38m14.807632627s) [upload=14034]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:01.235662578 (13h38m22.979719439s) [upload=14043]: Starting upload Aug 7 23:18:01.235769067 (13h38m22.97982594s) [upload=14043]: Done starting upload Aug 7 23:18:01.235873103 (13h38m22.979929971s) [upload=14043]: Uploading segments... Aug 7 23:18:01.235955349 (13h38m22.98001222s) [upload=14043]: Got next segment. Inline: true Aug 7 23:18:01.235976122 (13h38m22.980032984s) [upload=14043]: Waiting for error group managing segments... Aug 7 23:18:01.236032940 (13h38m22.980089801s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:01.236038848 (13h38m22.980095712s) [upload=14043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:01.236056240 (13h38m22.980113107s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:30.962606072 (13h38m52.706662932s) [upload=14043]: Done uploading segments... Aug 7 23:18:30.962671417 (13h38m52.706728279s) [upload=14043]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:52.160899256 (13h39m13.904956117s) [upload=14060]: Starting upload Aug 7 23:18:52.160970297 (13h39m13.905027157s) [upload=14060]: Done starting upload Aug 7 23:18:52.161065587 (13h39m13.905122439s) [upload=14060]: Uploading segments... Aug 7 23:18:52.161108065 (13h39m13.905164934s) [upload=14060]: Got next segment. Inline: true Aug 7 23:18:52.161113994 (13h39m13.905170844s) [upload=14060]: Waiting for error group managing segments... Aug 7 23:18:52.161171289 (13h39m13.90522814s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:52.161176753 (13h39m13.905233614s) [upload=14060]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:52.161180292 (13h39m13.905237155s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:52.299645097 (13h39m14.043701961s) [upload=14060]: Done uploading segments... Aug 7 23:18:52.299672156 (13h39m14.043729007s) [upload=14060]: Upload finished. err: <nil> Aug 7 23:19:29.899969190 (13h39m51.64402605s) [upload=14071]: Starting upload Aug 7 23:19:29.900023963 (13h39m51.644080812s) [upload=14071]: Done starting upload Aug 7 23:19:29.900097059 (13h39m51.644153908s) [upload=14071]: Uploading segments... Aug 7 23:19:29.900152738 (13h39m51.644209608s) [upload=14071]: Got next segment. Inline: true Aug 7 23:19:29.900161132 (13h39m51.644218002s) [upload=14071]: Waiting for error group managing segments... Aug 7 23:19:29.900212688 (13h39m51.64426955s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:29.900219311 (13h39m51.644276178s) [upload=14071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:29.900224731 (13h39m51.64428173s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:30.208671256 (13h39m51.952728106s) [upload=14071]: Done uploading segments... Aug 7 23:19:30.208715246 (13h39m51.952772096s) [upload=14071]: Upload finished. err: <nil> Aug 7 23:20:17.640185376 (13h40m39.384242236s) [upload=14084]: Starting upload Aug 7 23:20:17.640240796 (13h40m39.384297642s) [upload=14084]: Done starting upload Aug 7 23:20:17.640310076 (13h40m39.384366923s) [upload=14084]: Uploading segments... Aug 7 23:20:17.640362556 (13h40m39.384419425s) [upload=14084]: Got next segment. Inline: true Aug 7 23:20:17.640367233 (13h40m39.384424093s) [upload=14084]: Waiting for error group managing segments... Aug 7 23:20:17.640417842 (13h40m39.384474688s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:17.640422768 (13h40m39.384479628s) [upload=14084]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:20:17.640425589 (13h40m39.384482436s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:21.993094963 (13h40m43.737151826s) [upload=14084]: Done uploading segments... Aug 7 23:20:21.993114705 (13h40m43.737171555s) [upload=14084]: Upload finished. err: <nil> Aug 7 23:21:07.375939272 (13h41m29.119996122s) [upload=14096]: Starting upload Aug 7 23:21:07.376022371 (13h41m29.120079233s) [upload=14096]: Done starting upload Aug 7 23:21:07.376131644 (13h41m29.120188494s) [upload=14096]: Uploading segments... Aug 7 23:21:07.376228707 (13h41m29.120285578s) [upload=14096]: Got next segment. Inline: true Aug 7 23:21:07.376236415 (13h41m29.120293285s) [upload=14096]: Waiting for error group managing segments... Aug 7 23:21:07.376314696 (13h41m29.120371567s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:21:07.376322308 (13h41m29.120379199s) [upload=14096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:21:07.376327371 (13h41m29.120384242s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:37.104686329 (13h41m58.84874319s) [upload=14096]: Done uploading segments... Aug 7 23:21:37.104719076 (13h41m58.848775939s) [upload=14096]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:22:04.330992472 (13h42m26.075049325s) [upload=14111]: Starting upload Aug 7 23:22:04.331290477 (13h42m26.075347342s) [upload=14111]: Done starting upload Aug 7 23:22:04.331454134 (13h42m26.075510996s) [upload=14111]: Uploading segments... Aug 7 23:22:04.331559270 (13h42m26.075616144s) [upload=14111]: Got next segment. Inline: true Aug 7 23:22:04.331568404 (13h42m26.075625271s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:22:04.331647006 (13h42m26.075703873s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:22:04.331655756 (13h42m26.075712624s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:22:04.331661330 (13h42m26.075718197s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:22:04.632273425 (13h42m26.376330276s) [upload=14111]: Done uploading segments... Aug 7 23:22:04.632293274 (13h42m26.376350128s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:22:37.291949339 (13h42m59.03600619s) [upload=14122]: Starting upload Aug 7 23:22:37.292013250 (13h42m59.036070113s) [upload=14122]: Done starting upload Aug 7 23:22:37.292085244 (13h42m59.036142097s) [upload=14122]: Uploading segments... Aug 7 23:22:37.292207393 (13h42m59.036264263s) [upload=14122]: Got next segment. Inline: true Aug 7 23:22:37.292216268 (13h42m59.036273132s) [upload=14122]: Waiting for error group managing segments... Aug 7 23:22:37.292274925 (13h42m59.036331788s) [upload=14122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:22:37.292281983 (13h42m59.036338843s) [upload=14122]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:22:37.292288349 (13h42m59.036345217s) [upload=14122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:23:07.017248949 (13h43m28.761305819s) [upload=14122]: Done uploading segments... Aug 7 23:23:07.017274459 (13h43m28.761331331s) [upload=14122]: Upload finished. err: metaclient: context deadline exceeded Duration: 985.114199ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........................................XXXXXX.XX........] duration: 2.549872727s FAIL (2025-08-07 23:26:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.549919561s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X.XX..X...........] duration: 2.672527534s FAIL (2025-08-07 23:25:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.672559143s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [............................................X.XXXXX.X.......] duration: 3.035047132s FAIL (2025-08-07 23:25:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.035076203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X....X............X...........................] duration: 3.303785025s FAIL (2025-08-07 23:25:58 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 3.303827725s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.169299996s FAIL (2025-08-07 23:25:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.169328846s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.......................................XXX.XXXXXX..XX.X.....] duration: 856.155199ms FAIL (2025-08-07 23:25:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.340430421 (13h37m45.084487291s) [upload=14034]: Starting upload Aug 7 23:17:23.340585398 (13h37m45.084642265s) [upload=14034]: Done starting upload Aug 7 23:17:23.340743782 (13h37m45.084800644s) [upload=14034]: Uploading segments... Aug 7 23:17:23.340847668 (13h37m45.084904539s) [upload=14034]: Got next segment. Inline: true Aug 7 23:17:23.340857086 (13h37m45.084913958s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:17:23.340963706 (13h37m45.085020572s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.340973939 (13h37m45.085031006s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.340979371 (13h37m45.08503624s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.063547301 (13h38m14.807604164s) [upload=14034]: Done uploading segments... Aug 7 23:17:53.063575766 (13h38m14.807632627s) [upload=14034]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:01.235662578 (13h38m22.979719439s) [upload=14043]: Starting upload Aug 7 23:18:01.235769067 (13h38m22.97982594s) [upload=14043]: Done starting upload Aug 7 23:18:01.235873103 (13h38m22.979929971s) [upload=14043]: Uploading segments... Aug 7 23:18:01.235955349 (13h38m22.98001222s) [upload=14043]: Got next segment. Inline: true Aug 7 23:18:01.235976122 (13h38m22.980032984s) [upload=14043]: Waiting for error group managing segments... Aug 7 23:18:01.236032940 (13h38m22.980089801s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:01.236038848 (13h38m22.980095712s) [upload=14043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:01.236056240 (13h38m22.980113107s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:30.962606072 (13h38m52.706662932s) [upload=14043]: Done uploading segments... Aug 7 23:18:30.962671417 (13h38m52.706728279s) [upload=14043]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:52.160899256 (13h39m13.904956117s) [upload=14060]: Starting upload Aug 7 23:18:52.160970297 (13h39m13.905027157s) [upload=14060]: Done starting upload Aug 7 23:18:52.161065587 (13h39m13.905122439s) [upload=14060]: Uploading segments... Aug 7 23:18:52.161108065 (13h39m13.905164934s) [upload=14060]: Got next segment. Inline: true Aug 7 23:18:52.161113994 (13h39m13.905170844s) [upload=14060]: Waiting for error group managing segments... Aug 7 23:18:52.161171289 (13h39m13.90522814s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:52.161176753 (13h39m13.905233614s) [upload=14060]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:52.161180292 (13h39m13.905237155s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:52.299645097 (13h39m14.043701961s) [upload=14060]: Done uploading segments... Aug 7 23:18:52.299672156 (13h39m14.043729007s) [upload=14060]: Upload finished. err: <nil> Aug 7 23:19:29.899969190 (13h39m51.64402605s) [upload=14071]: Starting upload Aug 7 23:19:29.900023963 (13h39m51.644080812s) [upload=14071]: Done starting upload Aug 7 23:19:29.900097059 (13h39m51.644153908s) [upload=14071]: Uploading segments... Aug 7 23:19:29.900152738 (13h39m51.644209608s) [upload=14071]: Got next segment. Inline: true Aug 7 23:19:29.900161132 (13h39m51.644218002s) [upload=14071]: Waiting for error group managing segments... Aug 7 23:19:29.900212688 (13h39m51.64426955s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:29.900219311 (13h39m51.644276178s) [upload=14071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:29.900224731 (13h39m51.64428173s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:30.208671256 (13h39m51.952728106s) [upload=14071]: Done uploading segments... Aug 7 23:19:30.208715246 (13h39m51.952772096s) [upload=14071]: Upload finished. err: <nil> Aug 7 23:20:17.640185376 (13h40m39.384242236s) [upload=14084]: Starting upload Aug 7 23:20:17.640240796 (13h40m39.384297642s) [upload=14084]: Done starting upload Aug 7 23:20:17.640310076 (13h40m39.384366923s) [upload=14084]: Uploading segments... Aug 7 23:20:17.640362556 (13h40m39.384419425s) [upload=14084]: Got next segment. Inline: true Aug 7 23:20:17.640367233 (13h40m39.384424093s) [upload=14084]: Waiting for error group managing segments... Aug 7 23:20:17.640417842 (13h40m39.384474688s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:17.640422768 (13h40m39.384479628s) [upload=14084]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:20:17.640425589 (13h40m39.384482436s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:21.993094963 (13h40m43.737151826s) [upload=14084]: Done uploading segments... Aug 7 23:20:21.993114705 (13h40m43.737171555s) [upload=14084]: Upload finished. err: <nil> Aug 7 23:21:07.375939272 (13h41m29.119996122s) [upload=14096]: Starting upload Aug 7 23:21:07.376022371 (13h41m29.120079233s) [upload=14096]: Done starting upload Aug 7 23:21:07.376131644 (13h41m29.120188494s) [upload=14096]: Uploading segments... Aug 7 23:21:07.376228707 (13h41m29.120285578s) [upload=14096]: Got next segment. Inline: true Aug 7 23:21:07.376236415 (13h41m29.120293285s) [upload=14096]: Waiting for error group managing segments... Aug 7 23:21:07.376314696 (13h41m29.120371567s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:21:07.376322308 (13h41m29.120379199s) [upload=14096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:21:07.376327371 (13h41m29.120384242s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:37.104686329 (13h41m58.84874319s) [upload=14096]: Done uploading segments... Aug 7 23:21:37.104719076 (13h41m58.848775939s) [upload=14096]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:22:04.330992472 (13h42m26.075049325s) [upload=14111]: Starting upload Aug 7 23:22:04.331290477 (13h42m26.075347342s) [upload=14111]: Done starting upload Aug 7 23:22:04.331454134 (13h42m26.075510996s) [upload=14111]: Uploading segments... Aug 7 23:22:04.331559270 (13h42m26.075616144s) [upload=14111]: Got next segment. Inline: true Aug 7 23:22:04.331568404 (13h42m26.075625271s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:22:04.331647006 (13h42m26.075703873s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:22:04.331655756 (13h42m26.075712624s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:22:04.331661330 (13h42m26.075718197s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:22:04.632273425 (13h42m26.376330276s) [upload=14111]: Done uploading segments... Aug 7 23:22:04.632293274 (13h42m26.376350128s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:22:37.291949339 (13h42m59.03600619s) [upload=14122]: Starting upload Aug 7 23:22:37.292013250 (13h42m59.036070113s) [upload=14122]: Done starting upload Aug 7 23:22:37.292085244 (13h42m59.036142097s) [upload=14122]: Uploading segments... Aug 7 23:22:37.292207393 (13h42m59.036264263s) [upload=14122]: Got next segment. Inline: true Aug 7 23:22:37.292216268 (13h42m59.036273132s) [upload=14122]: Waiting for error group managing segments... Aug 7 23:22:37.292274925 (13h42m59.036331788s) [upload=14122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:22:37.292281983 (13h42m59.036338843s) [upload=14122]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:22:37.292288349 (13h42m59.036345217s) [upload=14122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:23:07.017248949 (13h43m28.761305819s) [upload=14122]: Done uploading segments... Aug 7 23:23:07.017274459 (13h43m28.761331331s) [upload=14122]: Upload finished. err: metaclient: context deadline exceeded Duration: 856.185897ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [.........................................XX..XXXXX..........] duration: 975.33374ms FAIL (2025-08-07 23:25:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:11:40.667902567 (13h32m2.411959413s) [upload=13933]: Starting upload Aug 7 23:11:40.667961940 (13h32m2.412018788s) [upload=13933]: Done starting upload Aug 7 23:11:40.668020524 (13h32m2.41207737s) [upload=13933]: Uploading segments... Aug 7 23:11:40.668075247 (13h32m2.412132113s) [upload=13933]: Got next segment. Inline: true Aug 7 23:11:40.668080664 (13h32m2.412137525s) [upload=13933]: Waiting for error group managing segments... Aug 7 23:11:40.668121190 (13h32m2.412178036s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:40.668125756 (13h32m2.412182616s) [upload=13933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:40.668128431 (13h32m2.412185291s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.394243254 (13h33m2.138300118s) [upload=13933]: Done uploading segments... Aug 7 23:12:40.394270270 (13h33m2.138327132s) [upload=13933]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:40.798630406 (13h33m2.542687267s) [upload=13953]: Starting upload Aug 7 23:12:40.798725809 (13h33m2.54278266s) [upload=13953]: Done starting upload Aug 7 23:12:40.798811946 (13h33m2.542868796s) [upload=13953]: Uploading segments... Aug 7 23:12:40.798867069 (13h33m2.542923932s) [upload=13953]: Got next segment. Inline: true Aug 7 23:12:40.798873108 (13h33m2.542929959s) [upload=13953]: Waiting for error group managing segments... Aug 7 23:12:40.798918573 (13h33m2.542975422s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:40.798924603 (13h33m2.542981463s) [upload=13953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:40.798939855 (13h33m2.542996716s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.942390421 (13h33m2.686447272s) [upload=13953]: Done uploading segments... Aug 7 23:12:40.942410212 (13h33m2.686467062s) [upload=13953]: Upload finished. err: <nil> Aug 7 23:13:18.137108974 (13h33m39.881165825s) [upload=13967]: Starting upload Aug 7 23:13:18.137180408 (13h33m39.881237292s) [upload=13967]: Done starting upload Aug 7 23:13:18.137258925 (13h33m39.881315786s) [upload=13967]: Uploading segments... Aug 7 23:13:18.137309635 (13h33m39.881366505s) [upload=13967]: Got next segment. Inline: true Aug 7 23:13:18.137314511 (13h33m39.881371371s) [upload=13967]: Waiting for error group managing segments... Aug 7 23:13:18.137374772 (13h33m39.881431633s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.137380412 (13h33m39.881437273s) [upload=13967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.137383733 (13h33m39.881440593s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.301480519 (13h33m40.045537379s) [upload=13967]: Done uploading segments... Aug 7 23:13:18.301498918 (13h33m40.045555767s) [upload=13967]: Upload finished. err: <nil> Aug 7 23:14:13.169029066 (13h34m34.913085913s) [upload=13980]: Starting upload Aug 7 23:14:13.169095254 (13h34m34.913152101s) [upload=13980]: Done starting upload Aug 7 23:14:13.169153338 (13h34m34.913210184s) [upload=13980]: Uploading segments... Aug 7 23:14:13.169225662 (13h34m34.913282508s) [upload=13980]: Got next segment. Inline: true Aug 7 23:14:13.169230885 (13h34m34.913287744s) [upload=13980]: Waiting for error group managing segments... Aug 7 23:14:13.169272521 (13h34m34.913329381s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:13.169288952 (13h34m34.913345813s) [upload=13980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:13.169291623 (13h34m34.913348483s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:12.895960606 (13h35m34.640017467s) [upload=13980]: Done uploading segments... Aug 7 23:15:12.895983608 (13h35m34.640040456s) [upload=13980]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:13.308146067 (13h35m35.052202918s) [upload=13997]: Starting upload Aug 7 23:15:13.308219271 (13h35m35.052276134s) [upload=13997]: Done starting upload Aug 7 23:15:13.308297319 (13h35m35.052354171s) [upload=13997]: Uploading segments... Aug 7 23:15:13.308360203 (13h35m35.052417072s) [upload=13997]: Got next segment. Inline: true Aug 7 23:15:13.308365548 (13h35m35.052422408s) [upload=13997]: Waiting for error group managing segments... Aug 7 23:15:13.308409794 (13h35m35.052466657s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:13.308426102 (13h35m35.052482963s) [upload=13997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:13.308429725 (13h35m35.052486585s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:13.040589970 (13h36m34.784646843s) [upload=13997]: Done uploading segments... Aug 7 23:16:13.040614428 (13h36m34.784671274s) [upload=13997]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:20.767866726 (13h36m42.511923577s) [upload=14018]: Starting upload Aug 7 23:16:20.767927544 (13h36m42.511984398s) [upload=14018]: Done starting upload Aug 7 23:16:20.767993090 (13h36m42.512049942s) [upload=14018]: Uploading segments... Aug 7 23:16:20.768055054 (13h36m42.512111926s) [upload=14018]: Got next segment. Inline: true Aug 7 23:16:20.768061500 (13h36m42.512118351s) [upload=14018]: Waiting for error group managing segments... Aug 7 23:16:20.768130604 (13h36m42.512187577s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:20.768139327 (13h36m42.512196192s) [upload=14018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:20.768144987 (13h36m42.512201849s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:20.496845379 (13h37m42.240902231s) [upload=14018]: Done uploading segments... Aug 7 23:17:20.496873778 (13h37m42.240930632s) [upload=14018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:20.916777280 (13h37m42.660834146s) [upload=14032]: Starting upload Aug 7 23:17:20.916877286 (13h37m42.660934149s) [upload=14032]: Done starting upload Aug 7 23:17:20.917002557 (13h37m42.661059408s) [upload=14032]: Uploading segments... Aug 7 23:17:20.917058206 (13h37m42.661115077s) [upload=14032]: Got next segment. Inline: true Aug 7 23:17:20.917064317 (13h37m42.661121178s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:17:20.917116347 (13h37m42.661173214s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:20.917123150 (13h37m42.661180014s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:20.917127779 (13h37m42.661184648s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:20.640884349 (13h38m42.384941214s) [upload=14032]: Done uploading segments... Aug 7 23:18:20.640914749 (13h38m42.384971617s) [upload=14032]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:21.050365938 (13h38m42.79442281s) [upload=14051]: Starting upload Aug 7 23:18:21.050484720 (13h38m42.794541587s) [upload=14051]: Done starting upload Aug 7 23:18:21.050589902 (13h38m42.794646752s) [upload=14051]: Uploading segments... Aug 7 23:18:21.050663797 (13h38m42.794720667s) [upload=14051]: Got next segment. Inline: true Aug 7 23:18:21.050670451 (13h38m42.794727311s) [upload=14051]: Waiting for error group managing segments... Aug 7 23:18:21.050719413 (13h38m42.794776275s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:21.050725163 (13h38m42.794782023s) [upload=14051]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:21.050728952 (13h38m42.794785811s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:20.780076504 (13h39m42.524133366s) [upload=14051]: Done uploading segments... Aug 7 23:19:20.780099469 (13h39m42.52415633s) [upload=14051]: Upload finished. err: metaclient: context deadline exceeded Duration: 975.437981ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [.......................................XXXXX.XX.XX.X........] duration: 879.533425ms FAIL (2025-08-07 23:25:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 879.563365ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXX.......] duration: 3.00413465s FAIL (2025-08-07 23:25:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== upload Aug 7 23:18:25.127322319 (13h38m46.871379183s) [upload=14052]: Done starting upload Aug 7 23:18:25.127477364 (13h38m46.871534223s) [upload=14052]: Uploading segments... Aug 7 23:18:25.127943505 (13h38m46.872000369s) [upload=14052]: Got next segment. Inline: false Aug 7 23:18:25.128022972 (13h38m46.87207984s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:25.128032328 (13h38m46.872089195s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:44.923639390 (13h39m6.667696251s) [seg_pos={0 0}, upload=14052]: Begin upload segment... Aug 7 23:18:44.923694548 (13h39m6.667751411s) [seg_pos={0 0}, upload=14052]: Done begin upload segment. Aug 7 23:18:44.923719227 (13h39m6.667776093s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.923742940 (13h39m6.667799806s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924062820 (13h39m6.66811968s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924231371 (13h39m6.668288235s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924310412 (13h39m6.668367275s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924552072 (13h39m6.668608933s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924795545 (13h39m6.66885246s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925037101 (13h39m6.669093961s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925293050 (13h39m6.669349911s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925526381 (13h39m6.669583247s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925876037 (13h39m6.669932897s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926278285 (13h39m6.670335152s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926413382 (13h39m6.670470243s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926732421 (13h39m6.670789284s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927153074 (13h39m6.671209983s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927375228 (13h39m6.671432108s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:45.586845311 (13h39m7.330902181s) [upload=14052]: Next returned nil segment Aug 7 23:18:45.586873064 (13h39m7.330929934s) [upload=14052]: Waiting for error group managing segments... Aug 7 23:18:45.727354327 (13h39m7.471411192s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.767876801 (13h39m7.511933648s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.792222311 (13h39m7.536279182s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.836090391 (13h39m7.580147242s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.847910967 (13h39m7.591967821s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.874060717 (13h39m7.618117569s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.882796683 (13h39m7.626853529s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.883366192 (13h39m7.62742305s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.899253981 (13h39m7.643310835s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.905112812 (13h39m7.64916966s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.911636868 (13h39m7.655693715s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.934683670 (13h39m7.67874052s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959619489 (13h39m7.703676335s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959647576 (13h39m7.703704436s) [seg_pos={0 0}, upload=14052]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:45.959785097 (13h39m7.703841946s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959873513 (13h39m7.703930376s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959997706 (13h39m7.704054568s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection Aug 7 23:18:45.960040133 (13h39m7.70409698s) [seg_pos={0 0}, upload=14052]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:45.960143346 (13h39m7.704200223s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:45.960151195 (13h39m7.704208074s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:34.514397794 (13h40m56.258454651s) [upload=14052]: Done uploading segments... Aug 7 23:20:34.514434819 (13h40m56.258491687s) [upload=14052]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:20:34.909564598 (13h40m56.65362148s) [upload=14088]: Starting upload Aug 7 23:20:34.909657672 (13h40m56.653714552s) [upload=14088]: Done starting upload Aug 7 23:20:34.909836458 (13h40m56.653893325s) [upload=14088]: Uploading segments... Aug 7 23:20:34.910062213 (13h40m56.654119064s) [upload=14088]: Got next segment. Inline: false Aug 7 23:20:34.910103244 (13h40m56.654160095s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:34.910108545 (13h40m56.654165405s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:04.641072069 (13h41m26.385128931s) [upload=14088]: Next segment err: context deadline exceeded Aug 7 23:21:04.641101566 (13h41m26.385158427s) [upload=14088]: Done uploading segments... Aug 7 23:21:04.641122347 (13h41m26.3851792s) [upload=14088]: Upload finished. err: context deadline exceeded Duration: 3.004169923s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)